Image processing apparatus and image processing method

ABSTRACT

An example image processor provides a distortion-corrected image including a masking area having improved appearance. The image processor includes: a masking area retaining section that retains a masking area set for an image having distortion; a masking execution section that performs masking processing for the image having distortion using the masking area retained in the masking area retaining section; a distortion correction section that corrects the image having distortion that is subjected to the masking processing using a correction parameter for correcting distortion of the image having distortion; and a masking area shaping section that shapes the masking area of the corrected image into a predetermined shape.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part of International ApplicationNo. PCT/JP2013/000423, filed Jan. 28, 2013, and also claims the benefitsof Patent Application No. 2012-017576 filed in Japan on Jan. 31, 2012,Patent Application No. 2012-027160 filed in Japan on Feb. 10, 2012,Patent Application No. 2012-047166 filed in Japan on Mar. 2, 2012, andPatent Application No. 2012-075741 filed in Japan on Mar. 29, 2012, thecontents of which are incorporated by reference herein in theirentirety.

FIELD

The technology herein relates to an image processing apparatus and animage forming method that performs distortion correction processing foran image having distortion and performs masking processing for a partialarea of the image having distortion.

BACKGROUND AND SUMMARY

Conventionally, monitoring cameras are often installed in public spacessuch as a street. An image obtained by shooting with the monitoringcamera may include an area unnecessary for monitoring purposes. Further,from a viewpoint of privacy protection, it may be undesirable to acquireand store a still image or a moving image of such an unnecessary area.In order to cope with this, there is known a shooting system that shootsa range to be monitored with a monitoring camera and performs maskingprocessing for a partial area of an image obtained by the monitoringcamera for privacy protection (for example, see Patent Literature 1 (JP3996805 B2) and Patent Literature 2 (U.S. Pat. No. 7,366,359)).

An omnidirectional camera provided with a fish-eye lens and thus capableof shooting an omnidirectional image is sometimes used as a monitoringcamera. An angle of view in such an omnidirectional camera is about360°, and an omnidirectional image having a circular shape or a doughnutshape is obtained. In this omnidirectional image, uneven distortionoccurs depending on location. In order to facilitate visualization ofthe circular or doughnut shaped omnidirectional image on a rectangularviewer screen, distortion correction processing is performed for theomnidirectional image. Specifically, for example, a partial area of theomnidirectional image is cut out, and then distortion correctionprocessing is performed for the cut-out image, or distortion correctionprocessing is performed for the omnidirectional image to develop theomnidirectional image into a panoramic image.

FIG. 41A illustrates an example of the omnidirectional image, FIG. 41Billustrates an example in which cut-out/distortion-corrected imagescorresponding to a plurality of cut-out ranges are displayed in ajuxtaposed manner, together with the omnidirectional image, and FIG. 41Cillustrates a display example of a panoramic image corresponding to theomnidirectional image of FIG. 41A. As illustrated in FIGS. 41A to 41C,when the omnidirectional image is obtained, a plurality of ways ofdistortion correction processing can be performed for theomnidirectional image.

As described above, the conventional monitoring camera can arbitrarilyspecify the cut-out range in the omnidirectional image, i.e., a cut-outview angle and a cut-out position and obtain acut-out/distortion-corrected image corresponding to the cut-out viewangle and position. Further, during shooting of a moving image, thecut-out range can be changed in real time. For example, a configurationis possible, in which the cut-out range is changed so as to follow amoving object detected in an image and, at the same time, thecut-out/distortion-corrected image corresponding to the cut-out range isgenerated.

When the masking processing is performed for the partial area on theomnidirectional image, distortion is removed from an unmasking area byperforming the distortion correction processing; however, the maskingarea assumes a distorted shape in the distortion-corrected image andlooks horrible.

In a monitoring camera system of Patent Literature 1, the masking areais displayed in a rectangle. FIG. 18 is a view for explaining processingperformed in Patent Literature 1. In the drawing, P1 to P4 are each avertex of the masking area, and MLU, MRU, MLD, and MRD are each a vertexof a rectangle formed by horizontal and vertical lines circumscribed toa figure surrounded by P1 to P4. In the monitoring camera system ofPatent Literature 1, when the masking area surrounded by the P1 to P4 isdistorted in a case where a configuration of a signal generation sectionlimits the masking shape to a rectangle formed by horizontal andvertical lines, the area surrounded by the MLU, MRU, MLD, and MRD whichis circumscribed to the figure surrounded by P1 to P4 is masked.

The monitoring camera system of Patent Document 1 is only configured tomask the rectangle formed by horizontal and vertical lines circumscribedto the area to be masked which is surrounded by the P1 to P4 along therequirement of the signal generation section, but not configured todeform the shape of the masking area from a viewpoint of improvingappearance. That is, it is not always true that masking of the rectangleformed by the horizontal and vertical lines circumscribed to the maskingarea surrounded by P1 to P4 results in good visibility.

Further, it is difficult for a user to imagine how a shape of a maskingarea that the user specifies on the image having distortion changes onthe distortion-corrected image. Particularly, a correction parameter tobe used in distortion correction processing differs depending on thecut-out range, so that even if a masking area having the same shape andlocated at the same position as previous one is specified on the imagehaving distortion, when a plurality of different cut-out ranges is set,it is very difficult to estimate how the masking area is deformed by thedistortion correction processing to be performed for the individualcut-out range.

In a technology described in Patent Literature 2, a processing range isspecified on a distortion-corrected image, and the specified processingrange is projected onto a coordinate of the image having distortionbefore distortion correction processing. However, when the processingrange is specified by a method as described in Patent Literature 2, thatspecifies an arbitrary range on the image having distortion, a shape ofthe range changes depending on presence/absence of the distortioncorrection processing, a difference in display mode, or an image cut-outposition. This complicates the range specification to be conducted by auser, and a position of the specified range is difficult to understand.Further, in the technology described in Patent Literature 2, it isnecessary to project a large number of coordinates before and after thedistortion correction processing and, accordingly, an amount ofinformation to be processed becomes extremely large.

A purpose of the present technology is to provide an improved imageprocessing apparatus and an improved image processing method forproviding a distortion-corrected image including a masking area.

The present disclosure provides an image processing apparatus thatincludes: a masking area retaining section that retains a masking areaset for an image having distortion; a masking processing section thatperforms masking processing for the image having distortion using themasking area of the masking area retaining section; a distortioncorrection section that corrects the image having distortion that issubjected to the masking processing using a correction parameter forcorrecting distortion of the image having distortion; and a masking areashaping section that shapes the masking area of the corrected image intoa predetermined shape.

As described below, the present technology has other aspects. Therefore,the disclosure of the technology is intended to provide some of theaspects of the technology, and is not intended to limit the scope of thetechnology as described and claimed herein.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an outer appearance view of a camera in a first embodiment ofthe present technology.

FIG. 2 is a partially cut perspective view of the camera in the firstembodiment of the present technology.

FIG. 3 is an exploded perspective view of an optical unit in the firstembodiment of the present technology.

FIG. 4 is a block diagram illustrating a configuration of a shootingsystem according to the first embodiment of the present technology.

FIG. 5 is a view for explaining shaping of a masking area in the firstembodiment of the present technology.

FIG. 6 is a view for explaining a flow of processing performed in theshooting system in the first embodiment of the present technology.

FIG. 7 is a view illustrating processing performed in a camera and aviewer and data exchange therebetween in a modification of the firstembodiment of the present technology.

FIG. 8 is a view illustrating processing performed in the camera andviewer and data exchange therebetween in another modification of thefirst embodiment of the present technology.

FIG. 9 is a block diagram illustrating a configuration of a shootingsystem according to a second embodiment of the present technology.

FIG. 10 is a view for explaining a flow of processing performed in theshooting system in the second embodiment of the present technology.

FIG. 11 is a block diagram illustrating a configuration of a shootingsystem according to a third embodiment of the present technology.

FIG. 12 is a view for explaining a flow of processing performed in theshooting system in the third embodiment of the present technology.

FIG. 13A is a view for explaining a cut-out direction (radial direction)in a first modification of the shaping of the masking area according tothe embodiments of the present technology.

FIG. 13B is a view for explaining a cut-out direction (non-radialdirection) in the first modification of the shaping of the masking areaaccording to the embodiments of the present technology.

FIG. 14 is a view for explaining the shaping of the masking areas in thefirst modification of the shaping of the masking area according to theembodiments of the present technology.

FIG. 15 is a view for explaining the shaping of the masking areas in asecond modification of the shaping of the masking area according to theembodiments of the present technology.

FIG. 16 is a view for explaining the shaping of the masking areas in thesecond modification of the shaping of the masking area according to theembodiments of the present technology.

FIG. 17 is a view for explaining the shaping of the masking areas in thesecond modification of the shaping of the masking area according to theembodiments of the present technology.

FIG. 18 is a view for explaining processing to be performed for a maskedarea in a conventional approach.

FIG. 19 is a view illustrating an example of a preview image accordingto a fourth embodiment of the present technology.

FIG. 20 is a block diagram illustrating a shooting system according to afourth embodiment of the present technology.

FIG. 21 is a view illustrating an example of an omnidirectional imageaccording to the fourth embodiment of the present technology.

FIG. 22 is a view illustrating an example of masking area specificationaccording to the fourth embodiment of the present technology.

FIG. 23 is a view illustrating an example of masking area specificationaccording to the fourth embodiment of the present technology.

FIG. 24 is a view illustrating a cut-out/distortion-corrected imageaccording to the fourth embodiment of the present technology.

FIG. 25 is a flow diagram for explaining a procedure of generating thepreview image in the fourth embodiment of the present technology.

FIG. 26 is an operational flow diagram of the shooting system accordingto the fourth embodiment of the present technology.

FIG. 27 is a block diagram of a shooting system according to a fifthembodiment of the present technology.

FIG. 28 is a flow diagram for explaining a procedure of generating thepreview image in the fifth embodiment of the present technology.

FIG. 29 is an operational flow diagram of the shooting system accordingto the fifth embodiment of the present technology.

FIG. 30 is a block diagram of shooting system according to a sixthembodiment of the present technology.

FIG. 31 is a flow diagram for explaining a procedure of generating thepreview image in the sixth embodiment of the present technology.

FIG. 32 is an operational flow diagram of the shooting system accordingto the sixth embodiment of the present technology.

FIG. 33A is a view illustrating an example in which the masking area isspecified on the omnidirectional image in the modification of thepresent technology.

FIG. 33B is a view illustrating an example in which the cut-out range isspecified on the omnidirectional image in the modification of thepresent technology.

FIG. 33C is a view illustrating an example in which an instruction ofdisplay of the preview image on the omnidirectional image is issued inthe modification of the present technology.

FIG. 33D is a view illustrating a display example of the preview imagein the modification of the present technology.

FIG. 34 is a flowchart illustrating operation of a shooting systemaccording to a seventh embodiment of the present technology.

FIG. 35 is a block diagram of a shooting system according to the seventhembodiment of the present technology.

FIG. 36A is a view illustrating a processing range (when stored as a setof coordinates) stored in a range storage section according to theseventh embodiment of the present technology.

FIG. 36B is a view illustrating a processing range (when stored as a setof functions) stored in the range storage section according to theseventh embodiment of the present technology.

FIG. 37A is a view for explaining specification of a start point of theprocessing range made by a user in the seventh embodiment of the presenttechnology.

FIG. 37B is a view for explaining specification of an end point of theprocessing range made by a user in the seventh embodiment of the presenttechnology.

FIG. 38A is a view illustrating the start point and end point of theprocessing range specified on the omnidirectional image in the seventhembodiment of the present technology.

FIG. 38B is a view illustrating a display range determined based onspecification of the processing range in the seventh embodiment of thepresent technology.

FIG. 39 is a view illustrating the processing range displayed in asuperimposed manner on the preview image in the seventh embodiment ofthe present technology.

FIG. 40A is a view illustrating a display example of acut-out/distortion-corrected image in a normal mode in the seventhembodiment of the present technology.

FIG. 40B is a view illustrating a display example of a panoramic imagein the normal mode in the seventh embodiment of the present technology.

FIG. 41A is a view illustrating an example of the conventionalomnidirectional image.

FIG. 41B is a view illustrating an example in which cut-out imagescorresponding to a plurality of cut-out ranges are displayed in ajuxtaposed manner, together with the conventional omnidirectional image.

FIG. 41C is a view illustrating a display example of a panoramic imagecorresponding to the conventional omnidirectional image.

FIG. 42 is a block diagram illustrating a configuration of a shootingsystem according to an eighth embodiment of the present technology.

FIG. 43 is a flowchart illustrating processing in a shooting systemaccording to the eighth embodiment of the present technology.

FIG. 44 is a flowchart illustrating processing in a range generationsection according to the eighth embodiment.

FIG. 45 is a flowchart illustrating processing of generating positioninformation in the eighth embodiment of the present technology.

FIG. 46 is a view illustrating an example of image processing in theeighth embodiment of the present technology.

FIG. 47A is a view illustrating an example of range informationaccording to the eighth embodiment of the present technology.

FIG. 47B is a view illustrating an example of range informationaccording to the eighth embodiment of the present technology.

FIG. 48 is a view illustrating an example of specification of adirection of a shape in the eighth embodiment of the present technology.

FIG. 49A is a view illustrating an example of the conventionalomnidirectional image.

FIG. 49B is a view illustrating an example in which cut-out imagescorresponding to a plurality of cut-out ranges are displayed in ajuxtaposed manner, together with the conventional omnidirectional image.

FIG. 49C is a view illustrating a display example of a panoramic imagecorresponding to the conventional omnidirectional image.

DESCRIPTION OF EMBODIMENTS

Detailed description of the present technology will be described below.Embodiments to be described below are merely examples of the presenttechnology and can be modified into various aspects. Thus, specificstructures and functions disclosed herein do not limit the scope of thepresent technology.

An image processing apparatus according to one embodiment of the presenttechnology includes: a masking area retaining section that retains amasking area set for an image having distortion; a masking processingsection that performs masking processing for the image having distortionusing the masking area of the masking area retaining section; adistortion correction section that corrects the image having distortionthat is subjected to the masking processing using a correction parameterfor correcting distortion of the image having distortion; and a maskingarea shaping section that shapes the masking area of the corrected imageinto a predetermined shape.

According to this configuration, appearance of the masking area on adistortion-corrected image obtained by correcting the distortion of theimage having distortion is improved.

The image processing apparatus may include a masking area detectionsection that detects the masking area of the image corrected by thedistortion correction section. The masking area shaping section mayshape the masking area detected by the masking area detection sectioninto a predetermined shape.

According to this configuration, the masking area detection section candetect the masking area from the distortion-corrected image and canthereby shape the detected masking area.

In the image processing apparatus, the masking area shaping section mayshape the masking area into a rectangle surrounded by linesperpendicular and parallel to the image.

According to this configuration, the masking area can be shaped into ashape having improved appearance.

In the image processing apparatus, the masking area shaping section mayshape the masking area such that a line that is transformed into a curveby distortion correction processing by the distortion correction sectionis transformed into a straight line.

According to this configuration, the masking area can be shaped into ashape having improved natural appearance.

In the image processing apparatus, the image having distortion may be animage cut out from an omnidirectional image, and the masking areashaping section may shape the masking area according to a direction inwhich the image having distortion is cut out from the omnidirectionalimage.

According to this configuration, the masking area can be shaped suchthat an upright direction of a subject and that of the masking areacoincide with each other on an image cut out from the omnidirectionalimage.

In the image processing apparatus, in performing the correctionprocessing, the distortion correction section may linearly interpolatesource coordinates for each square formed by a plurality of pixels tofill in pixels of corresponding coordinates, and the masking areashaping section may set the square including the corrected masking areaas a masking area after shaping.

According to this configuration, the masking area is not formed into anarea including a diagonal or a curved side but can be formed into ashape along a boundary of the square, thereby improving appearance ofthe masking area.

In the image processing apparatus, the masking area shaping section mayset a plurality of squares including the squares that include thecorrected masking area and formed into a rectangular shape as a whole asa masking area after shaping.

According to this configuration, the masking area can be shaped usingthe square used in the distortion correction processing, therebyimproving appearance of the masking area.

The image processing apparatus may include an imaging section thatinputs the image having distortion.

According to this configuration, appearance of the masking area on thedistortion-corrected image obtained by correcting the distortion of theimage having distortion input from the shooting section is improved.

In the image processing apparatus, the imaging section may include afish-eye lens.

According to this configuration, an image having distortion with a wideview angle can be obtained in the shooting section, and appearance ofthe masking area on the distortion-corrected image obtained bycorrecting the distortion of the image having distortion is improved.

The image processing apparatus may include: a specification section thata user uses to specify, on the image having distortion, a masking areato be retained in the masking area retaining section; and a masking areacorrection section that corrects, by the correction parameter, themasking area specified by the specification section. The correctedmasking area may be displayed simultaneously when the user specifies themasking area.

According to this configuration, when specifying a masking area, it ispossible to confirm how the specified masking area is applied to theimage having distortion.

The image processing apparatus may further include a specificationsection that a user uses to specify, on the image having distortion, amasking area to be retained in the masking area retaining section; and amasking area correction section that corrects, by the correctionparameter, the masking area specified by the specification section. Thecorrected masking area may be displayed in a superimposed manner on theimage having distortion when the user specifies the masking area.

According to this configuration, it is possible to confirm a specifiedmasking area on the image having distortion.

An image processing apparatus according to another embodiment of thepresent technology includes: a masking area retaining section thatretains a masking area set for an image having distortion; a maskingarea correction section that corrects the masking area of the maskingarea retaining section using a correction parameter for correcting theimage having distortion; a masking area shaping section that shapes themasking area corrected by the masking area correction section into apredetermined shape; an inverse transform/correction section thatperforms inverse transform using the correction parameter for themasking area shaped by the masking area shaping section; a maskingprocessing section that performs masking processing for the image havingdistortion using the masking area that is subjected to the inversetransform by the inverse transform/correction section; and a distortioncorrection section that corrects the image having distortion that issubjected to the masking processing using the correction parameter.

According to this configuration, it is possible to shape the maskingarea to be subjected to the correction processing into a shape with goodvisibility after the distortion correction processing to be performedfor the image having distortion on which the masking processing issubjected to the masking area.

An image processing apparatus according to still another embodiment ofthe present technology includes: a masking area retaining section thatretains a masking area set for an image having distortion; a distortioncorrection section that corrects the image having distortion using acorrection parameter for correcting distortion of the image havingdistortion; a masking area correction section that corrects the maskingarea of the masking area retaining section using the correctionparameter; a masking area shaping section that shapes the masking areacorrected by the masking area correction section into a predeterminedshape; and a masking processing section that performs masking processingfor the masking area on the image corrected by the distortion correctionsection which is shaped by the masking area shaping section.

According to this configuration, both the image having distortion andmasking area are corrected, and the masking area is then shaped into ashape with improved appearance to be applied to the distortion-correctedimage.

An image processing method according to one embodiment of the presenttechnology includes the steps of: acquiring an image having distortion;retaining a masking area set for the image having distortion; performingmasking processing for the image having distortion using the maskingarea; correcting the image having distortion that is subjected to themasking processing using a correction parameter for correctingdistortion of the image having distortion; and shaping the masking areaof the corrected image into a predetermined shape.

Also according to this configuration, appearance of the masking area ona distortion-corrected image obtained by correcting the distortion ofthe image having distortion is improved.

Hereinafter, a shooting system according to respective embodiments ofthe present technology will be described using the drawings. A shootingsystem described herein is a system that processes a shot image, and apart of or the entire configuration of the shooting system can beregarded as an image processing apparatus.

1. First to Third Embodiments 1-1. First Embodiment

First, a structure of a camera according to the present embodiment willbe described. FIG. 1 is an outer appearance view of the camera in thepresent embodiment. FIG. 2 is a partially cut perspective view of thecamera in the present embodiment. As illustrated in FIG. 1, a camera 100has a casing 11, a dome cover 12, and a base 13. The camera 100 is usedfor monitoring, e.g., facilities and installed on a wall or a ceiling ofthe facilities. The casing 11 formed into an annular shape withsynthetic resin is positioned at an outer periphery of the camera 100,and the dome cover 12 formed of transparent plastic is disposed at acenter of the camera 100. An outer surface side of the base 13 ismounted to the wall or ceiling by mounting brackets 18. In this case,the dome cover 12 faces downward.

As illustrated in FIG. 2, a component housing space is formed around anoptical unit 14 of the base 13, and a main board 15 for image processingand power supply is provided in the component housing space. Acontroller (not illustrated) provided with a CPU and a storage sectionis mounted on the main board 15. The optical unit 14 has a fish-eye lens16 supported by a lens holder 19 and an openable/closable cover 17.

FIG. 3 is an exploded perspective view of the optical unit in thepresent embodiment. In FIG. 3, the fish-eye lens 16 is fitted to thelens holder 19 through a packing material 20. A MOS holder 24 is fixedto a sensor board 21 connected to the main board 15 by fixing screws 23.The sensor substrate 21 to which the MOS holder 24 is fixed to a lowersurface side of the lens holder 19 by a MOS bracket 25 and fixing screws26.

Four through holes 22 are provided at an outer periphery of the lensholder 19. Four fitting bosses 29 are provided from the base 13 and fixsupport posts 28, respectively. The support posts 28 each penetrates thethrough hole 22 of the lens holder 19 and fixed by fixing screws 21. Acoil spring 27 is extrapolated between the base 13 and through holes 22of the lens holder 19. The coil spring 27 biases upward the lens holder19 relative to the base 13.

FIG. 4 is a block diagram illustrating a configuration of a shootingsystem according to the present embodiment. As illustrated in FIG. 4, ashooting system 101 according to the first embodiment includes a camera110 and a viewer 210. The camera 110 is a monitoring camera installed ata place to be monitored. The viewer 210 is communicably connected to thecamera 110 by wire or wireless. The viewer 210 is realized by, e.g., acomputer having a monitor provided in a monitoring room or a mobileterminal carried by a monitoring person.

The camera 110 has an omnidirectional imaging section 111, a maskingdata retaining section 112, a masking execution section 113, a cut-outparameter setting section 114, and a cut-out/distortion correctionsection 115. The omnidirectional imaging section 111 is a well-knownomnidirectional camera including a fish-eye lens, an imaging element onwhich light from a subject (monitoring target) traveling through thefish-eye lens, and a processing section that performs processing forimage signals obtained by the imaging element to generate image data.The omnidirectional imaging section 111 outputs an omnidirectional imagegenerated by shooting to the masking execution section 113.

The omnidirectional image is a circular or doughnut shaped imageobtained by the fish-eye lens and having a horizontal view angle ofabout 360°. The omnidirectional image includes distortion, and a degreeof the distortion increases with proximity to an outer periphery of theomnidirectional image. The viewer 210 can directly display such anomnidirectional image and, as described later, it can display an imagefrom which the distortion is removed.

The masking data retaining section 112 retains data (hereinafter,referred to as “masking data” or “masking area”) representing a maskingarea on the omnidirectional image. That is, the masking data retained bythe masking data retaining section 112 is data indicating a shape of themasking area, a direction thereof, and a position thereof on theomnidirectional image. A user can set an arbitrary area in theomnidirectional image as the masking area. That is, the user can set amasking area having an arbitrary shape (e.g., a rectangle, an ellipse, acircle, and the like) at an arbitrary position in the omnidirectionalimage and store the set masking area in the masking data retainingsection 112. A plurality of the masking areas may be set in theomnidirectional image.

The masking execution section 113 acquires the omnidirectional imagefrom the omnidirectional imaging section 111 and performs maskingprocessing for the masking area on which is set in the masking dataretaining section 112. The masking processing is processing of reducingan amount of information obtained from the image and includes processingof filling the masking area with a predetermined color to reduce theinformation amount of the masking area to zero, processing (mosaicprocessing) of transforming the masking area into an image having areduced resolution, processing of transforming the masking area into animage as if it is seen through an opaque glass, and the like. In thepresent embodiment, the processing of filling the masking area with onecolor is adopted.

The cut-out parameter setting section 114 sets a cut-out parameterindicating a cut-out range in the omnidirectional image. The cut-outparameter includes, e.g., center coordinates of the cut-out range, azoom factor (size of the cut-out range on the omnidirectional image),and a cut-out direction. The cut-out/distortion correction section 115cuts out the cut-out range identified by the cut-out parameter set bythe cut-out parameter setting section 114 from the omnidirectional imageon which the masking processing is performed for the massing area by themasking execution section 113 and performs correction so as to removedistortion to thereby generate a distortion-corrected image.

When the cut-out range is identified by the cut-out parameter settingsection 114, a correction parameter for removal of the distortion fromthe cot-out range is uniquely determined, so that the cut-out parameterset in the cut-out parameter setting section 114 serves as thecorrection parameter to be used in processing by the cut-out/distortioncorrection section 115. The cut-out parameter set in the cut-outparameter setting section 114 may be set by user's specification.Alternatively, the cut-out parameter may be set such that the cut-outrange includes an area including a predetermined subject recognized byimage recognition processing that the camera 110 itself performed forthe omnidirectional image.

The cut-out/distortion correction section 115 cuts out the cut-out rangeof the omnidirectional image on which the masking processing isperformed for the masking area and performs distortion correctionprocessing for the cut-out range to thereby generate adistortion-corrected image. The cut-out/distortion correction section115 performs the distortion correction processing for the cut-out rangeof the omnidirectional image to thereby correct the masking area appliedto the omnidirectional image. Upon generation of thedistortion-corrected image in the cut-out/distortion correction section115, the distortion-corrected image is transmitted to the viewer 210through a not illustrated interface. The viewer 210 receives thedistortion-corrected image.

The viewer 210 has a masking area detection section 211, a masking areashaping section 212, and an output section 213. The masking areadetection section 211 acquires the distortion-corrected imagetransmitted from the camera 110 and detects the masking area from thedistortion-corrected image. Since the masking area of the presentembodiment is filled with one color, the masking area detection section211 detects the corresponding area as the masking area. The detectedmasking area is output, together with the distortion-corrected image, tothe masking area shaping section 212.

The masking area shaping section 212 shapes the masking area detected bythe masking area detection section 211 into an easy-to-see shape by amethod to be described later and then performs once again the maskingprocessing on the distortion-corrected image. The output section 213 isa display device that displays the distortion-corrected image on whichthe second masking processing is performed by the masking area shapingsection 212.

Hereinafter, the shaping of the masking area by the masking area shapingsection 212 will be described. FIG. 5 is a view for explaining theshaping of the masking area by the masking area shaping section 212. Inthe example of FIG. 5, a rectangular masking area mo1 formed byhorizontal and vertical sides included in the omnidirectional image asan original image is set in the masking data retaining section 112.

The masking area mo1 is applied to the omnidirectional image by themasking execution section 113, and the cut-out range of thisomnidirectional image is corrected by the cut-out/distortion correctionsection 115. Then, the masking area mo1 is shaped into a masking areamm1 of FIG. 5. That is, the masking area applied to the omnidirectionalimage before the distortion correction processing assumes a rectangleformed by horizontal and vertical sides, i.e., a well-organized shape;however, after the distortion correction processing performed by thecut-out/distortion correction section 115, the masking area assumes adistorted quadrangle. The distortion-corrected masking area illustratedin FIG. 5 is just an example and, actually, the masking area assumes ashape determined by a distortion correction parameter determined by lensdistortion, and the like, and may assume not only the quadrangle butalso a shape including a curve.

The masking area shaping section 212 shapes the masking area assumingsuch a distorted quadrangle into a rectangle formed by horizontal andvertical sides including all or a part of the masking area. In FIG. 5, ashaped masking area ms11 assumes a rectangle formed by horizontal andvertical sides completely including the distorted quadrangle obtained asa result of the distortion correction processing, that is, having anarea larger than the distorted quadrangle, a shaped masking area ms12assumes a rectangle formed by horizontal and vertical sides including apart of the distorted quadrangle obtained as a result of the distortioncorrection processing. The masking area shaping section 212 performsonce again the masking processing for the area of the rectangle ms11 orms12.

In FIG. 5, a distortion-corrected masking area mm2 assumes a shape inwhich four sides are concave curves as a result of the distortioncorrection processing. The masking area shaping section 212 shapes themasking area mm2 such that the concave curves thereof each become astraight line. Further, in FIG. 5, a distortion-corrected masking areamm3 assumes a shape in which two opposing sides are concave curves andother two opposing sides are convex curves. The masking area shapingsection 212 shapes the masking area mm3 such that the concave curvesthereof each become a straight line and convex curves thereof eachbecome a straight line.

A flow of processing in the shooting system 101 having the aboveconfiguration will be described together with images and masking areas.FIG. 6 is a view for explaining the processing flow in the shootingsystem 101. The omnidirectional imaging section 111 shoots a place to bemonitored to acquire an omnidirectional image IM11 (step S61). Asillustrated in FIG. 6, the omnidirectional image IM11 is an image formedinto a circle, in which a degree of the distortion increases withproximity to an outer periphery thereof. The masking execution section113 performs, on the omnidirectional image IM11, the masking processingfor a masking area MA11 read out from the masking data retaining section112 (step S62). As a result, the omnidirectional image IM11 becomes animage IM12 on which the masking processing is performed for the maskingarea MA11.

The cut-out parameter (i.e., correction parameter) set by the maskingdata retaining section 112 is used to cut out the cut-out range from theimage IM12 that is partially subjected to the masking processing asdescribed above, and the distortion correction processing is performedfor the cut-out range (step S63). As a result, the cut-out range that issubjected to the distortion correction processing is cut out from theomnidirectional image IM12 on which the masking processing is performedto obtain a distortion-corrected image IM13. The distortion-correctedimage IM13 includes a masking area MA12. The masking area MA 12 is anarea resulting from the cut-out/distortion correction processing thatthe cut-out distortion correction section 115 performs for the originalmasking area MA11 and thus assumes a distorted shape.

The distortion-corrected image IM13 thus generated is transmitted fromthe camera 110 to the viewer 210 and then supplied to the masking areadetection section 211 of the viewer 210. The masking area detectionsection 211 detects the masking area from the distortion-corrected imageIM13 (step S64). When the masking area MA12 is detected, the maskingarea shaping section 212 shapes the masking area to obtain a shapedmasking area MA13 (step S65). In the example of FIG. 6, the shapedmasking area MA13 assumes a rectangle formed by horizontal and verticalsides surrounding the former distorted masking area MA12.

The masking area shaping section 212 performs the masking processing forthe shaped masking area MA13 on the distortion-corrected image IM13(step S66). The masking area shaping section 212 thus performs themasking processing for the rectangular masking area MA13 formed byhorizontal and vertical sides to thereby obtain a distortion-correctedimage IM14.

As described above, according to the first embodiment of the presenttechnology, the masking processing is performed on the omnidirectionalimage having distortion with a desired area being set as the maskingarea, and then the distortion correction processing is performed for theresultant image. By the distortion correction processing, the distortionof an image other than the masking area is eliminated; however, themasking area assumes a distorted shape. Then, the masking area detectionsection 211 detects the masking area from the distortion-correctedimage, and masking area shaping section 212 shapes the detected maskingarea into a shape with improved appearance, followed by the secondmasking processing. Thus, in the final distortion-corrected image, themasking area assumes a shape with improved appearance.

In the above-described first embodiment, the masking data retainingsection 112, masking execution section 113, cut-out parameter settingsection 114, and cut-out/distortion correction section 115 are providedin the camera 110, and the masking area detection section 211 andmasking area shaping section 212 are provided in the viewer 210;however, each of the above constituent elements may be provided in anyone of the camera 110 and viewer 210.

For example, the camera 110 may be provided with only theomnidirectional imaging section 111 and the viewer 210 may be providedwith all the constituent elements other than the omnidirectional imagingsection 111. FIG. 7 is a view illustrating processing performed in thecamera 110 and viewer 210 and data exchange therebetween in this case.In this case, the omnidirectional imaging section 111 of the camera 110performs imaging (step S71) to obtain an omnidirectional image andtransmits the obtained omnidirectional image to the viewer 210 (stepS72). In the viewer 210, the cut-out/distortion correction section 115performs cut-out and correction processing of cutting out the cut-outrange from the omnidirectional image and performing the distortioncorrection processing (step S73), and the output section 213 displaysthe distortion-corrected image (step S74). In FIG. 7, descriptions ofthe masking processing are omitted.

FIG. 8 is a view illustrating processing performed in the camera 110 andviewer 210 and data exchange therebetween in a case where the shootingsystem 101 having the configuration illustrated in FIG. 4 is modifiedsuch that the cut-out parameter setting section 114 is provided in theviewer 210. In this case, the omnidirectional imaging section 111 of thecamera 110 performs imaging (step S81), and the cut-out parametersetting section 114 of the viewer 210 specifies the cut-out range (stepS82). Then, the cut-out parameter identifying the cut-out range istransmitted from the viewer 210 to the camera 110 (step S83).

The cut-out/distortion correction section 115 of the camera 110 uses thecut-out parameter as the correction parameter to perform the distortioncorrection processing for the omnidirectional image (step S84) tothereby obtain the distortion-corrected image. The obtaineddistortion-corrected image is transmitted from the camera 110 to theviewer 210 (step S85) and displayed on the output section 213 (stepS86).

1-2. Second Embodiment

The following describes a second embodiment of the present technology.FIG. 9 is a block diagram illustrating a configuration of a shootingsystem according to a second embodiment of the present technology. Asillustrated in FIG. 9, a shooting system 102 according to the secondembodiment includes a camera 120 and a viewer 220. The same referencenumerals are given to the same constituent elements of the camera 120and viewer 220 as those of the camera 110 and viewer 210 of the firstembodiment, and detailed descriptions thereof will be omitted.

The camera 120 has an omnidirectional imaging section 111, a maskingdata retaining section 112, a masking execution section 113, acut-out/distortion correction section 115, and a masking area shapingsection 126. In the present embodiment, the masking area shaping section126 shapes the masking area read out from the masking data retainingsection 112 such that the masking area assumes a shape with improvedappearance as a result of the distortion correction processing that thecut-out/distortion correction section 115 performs therefor and suppliesthe resultant masking area to the masking execution section 113. Then,the masking execution section 113 performs, on the omnidirectionalimage, the masking processing for the previously shaped masking area,and the cut-out/distortion correction section 115 performscut-out/distortion correction processing for the omnidirectional imageon which the masking processing is performed for the masking area.

To this end, the camera 120 has the masking area shaping section 126between the masking data retaining section 112 and masking executionsection 113. The masking area shaping section 126 includes a correctedmasking area generation section 1261, a corrected masking area shapingsection 1262, and an inverse transform/correction section 1263. Thecut-out parameter set by the cut-out parameter setting section 114 ofthe viewer 220 is transmitted from the viewer 220 to the camera 120 andsupplied to the corrected masking area generation section 1261, inversetransform/correction section 1263, and cut-out/distortion correctionsection 115 of the camera 120.

The corrected masking area generation section 1261 reads out the maskingarea from the masking data retaining section 112 and uses the cut-outparameter as the correction parameter to correct the read out maskingarea to thereby generate a corrected masking area. The correctionapplied to the masking area is performed according the same method asthat used in the distortion correction processing that thecut-out/distortion correction section 115 performs using the samecut-out parameter. As described above, the masking data retainingsection 112 retains the rectangular masking area formed by horizontaland vertical sides. This rectangular masking area is subjected to theabove-described correction that the corrected masking area generationsection 1261 performs using the cut-out parameter, with the result thatthe corrected masking area assumes a distorted shape. The correctedmasking area is supplied to the corrected masking area shaping section1262.

The corrected masking area shaping section 1262 shapes the correctedmasking area into a shape having improved appearance. A specific methodfor the shaping is as described above. The shaped masking area issupplied to the inverse transform/correction section 1263. The inversetransform/correction section 1263 uses the cut-out parameter to performcorrection inverse to the correction performed by the corrected maskingarea generation section 1261 and cut-out/distortion correction section115. As a result, the masking area having improved appearance obtainedby the shaping in the corrected masking area generation section 1261assumes once again a distorted shape. This distorted shape is a shapethat will be made to return, by application of the distortion correctionprocessing using the cut-out parameter, to the improved appearance shapethat is previously obtained by the shaping in the corrected masking areageneration section 1261.

The masking area that is subjected to the inverse transform by theinverse transform/correction section 1263 is supplied to the maskingexecution section 113, and the masking execution section 113 performsthe masking processing for the supplied masking area on theomnidirectional image. The cut-out/distortion correction section 115performs the cut-out/distortion correction processing using the cut-outparameter for the thus obtained omnidirectional image on which themasking processing is performed for the masking area. As is clear fromFIG. 9, the cut-out parameter to be used as the correction parameter inthe cut-out/distortion correction section 115 is the same parameter asthe cut-out parameter used for the correction of the masking area in thecorrected masking area generation section 1261 and the cut-out parameterused for the inverse transform in the inverse transform/correctionsection 1263.

By the cut-out/distortion correction processing performed by thecut-out/distortion correction section 115, the masking area is made toassume a shape having improved appearance, and distortion of the areaother than the masking area is eliminated. The thus obtaineddistortion-corrected image on which the masking processing is performedis transmitted from the camera 120 to the viewer 220, and the viewer 220displays the distortion-corrected image on the output section 213.

A flow of processing in the shooting system 102 having the aboveconfiguration will be described together with images and masking areas.FIG. 10 is a view for explaining the processing flow in the shootingsystem 102. The corrected masking area generation section 1261 acquiresa masking area MA21 from the masking data retaining section 112 andacquires the cut-out parameter from the cut-out parameter settingsection 114. Then, the corrected masking area generation section 1261performs the distortion correction processing for the masking area MA21using the cut-out parameter as the correction parameter to therebygenerate a corrected masking area MA22 (step S101).

The corrected masking area shaping section 1262 shapes the correctedmasking area MA22 into a corrected masking area MA23 (step S102). Then,the inverse transform/correction section 1263 acquires the cut-outparameter from the cut-out parameter setting section 14 and uses thecut-out parameter as the correction parameter to perform, for thecorrected masking area MA23, transform inverse to the correctionperformed in step S102 (step S103) to thereby obtain an inversetransformed/corrected masking area MA24. The omnidirectional imagingsection 111 shoots a place to be monitored to acquire an omnidirectionalimage IM21 (step S104), and the masking execution section 113 uses theinverse transformed/corrected masking area MA24 obtained by the inversetransform/correction section 1263 to perform the masking processing onthe omnidirectional image IM21 (step S105), to thereby an image IM22 onwhich the masking processing is performed.

The cut-out/distortion correction section 115 performs, for the imageIM22 on which the masking processing is performed, thecut-out/distortion correction processing of the cut-out range using thecut-out parameter set by the cut-out parameter setting section 114 (stepS106) to thereby obtain a distortion-corrected image IM23. As describedabove, in the distortion-corrected image IM23, the masking processing isperformed for the masking area having improved appearance obtained bythe shaping of the corrected masking area shaping section 1262.

As described above, according to the second embodiment of the presenttechnology, correction using the same correction parameter as that usedin the cut-out/distortion correction processing performed for theomnidirectional image is performed for the masking area, with the resultthat the masking area assumes a distorted shape. Then, the distortedmasking area is shaped into a shape with improved appearance, which isonce inverse transformed by the correction parameter. Then, the maskingprocessing is performed on the omnidirectional image having distortionwith the area after the inversion transform used as the masking area.Finally, the cut-out/distortion correction section 115 uses the samecorrection parameter once again to perform the distortion correctionprocessing. As a result, in the thus obtained distortion-correctedimage, the masking area assumes a shape with improved appearance whichis once obtained by the shaping performed by the corrected masking areashaping section 1262.

In the second embodiment, as illustrated in FIG. 9, the viewer 220according to the present technology needs to include only the cut-outparameter setting section 114 and output section 213. Since processingloads of the cut-out parameter setting section 114 and output section213 are not so large, the viewer 220 can be realized even in a devicewith limited processing power. For example, the present embodiment iseffective for a case where the viewer 220 is realized by a mobileterminal.

1-3. Third Embodiment

The following describes a third embodiment of the present technology.FIG. 11 is a block diagram illustrating a configuration of a shootingsystem according to a third embodiment of the present technology. Asillustrated in FIG. 11, a shooting system 103 according to the thirdembodiment includes a camera 130 and a viewer 230. The same referencenumerals are given to the same constituent elements of the camera 130and viewer 230 as those of the camera 110 and viewer 210 of the firstembodiment, and detailed descriptions thereof will be omitted.

The camera 130 has an omnidirectional imaging section 111, a maskingdata retaining section 112, a masking execution section 113, acut-out/distortion correction section 115, a masking area distortioncorrection section 137, and a masking area shaping section 138.

The masking area distortion correction section 137 uses the cut-outparameter acquired from the cut-out parameter setting section 114 as thecorrection parameter to perform cut-out/distortion correction processingof the cut-out range for the masking area retained in the masking dataretaining section 112 to thereby generate a distortion-corrected maskingarea on the omnidirectional image. The distortion-corrected masking areais a masking area on the cut-out range and has a distorted shape. Thedistortion-corrected masking area is output to the masking area shapingsection 138.

The masking area shaping section 138 shapes the distortion-correctedmasking area on the cut-out range generated by the masking areadistortion correction section 137 into a shape having improvedappearance. A specific method for the shaping is as described above.

The cut-out/distortion correction section 115 performs thecut-out/distortion correction processing of the cut-out range for theomnidirectional image acquired by the omnidirectional imaging section111 using the cut-out parameter read out from the cut-out parametersetting section 114 as the correction parameter to thereby generate adistortion-corrected image. The masking execution section 113 inputs thedistortion-corrected image generated by the cut-out/distortioncorrection section 115 and the masking area shaped by the masking areashaping section 138 and performs the masking processing for the shapedmasking area on the distortion-corrected image.

The distortion-corrected image on which the masking processing isperformed by the masking execution section 113 is transmitted from thecamera 130 to the viewer 230. The output section 213 of the viewer 230displays the distortion-corrected image transmitted from the camera 130.

A flow of processing in the shooting system 103 having the aboveconfiguration will be described together with images and masking areas.FIG. 12 is a view for explaining the processing flow in the shootingsystem 103. The masking area distortion correction section 137 reads outthe cut-out parameter from the cut-out parameter setting section 114 anduses the read out cut-out parameter as the correction parameter tocorrect a masking area MA31 read out from the masking data retainingsection 112 (step S121) to thereby generate a corrected masking areaMA32. Here, the masking area distortion correction section 137 uses thecut-out parameter to cut the cut-out range from the omnidirectionalimage and then performs the distortion correction processing for themasking area MA31. Thereafter, the masking area shaping section 138shapes the masking area corrected by the masking area distortioncorrection section 137 into a shape having improved appearance (stepS122) to thereby generate a shaped masking area MA33.

The omnidirectional imaging section 111 shoots a monitoring target toacquire an omnidirectional image IM31 (step S123), and thecut-out/distortion correction section 115 uses the cut-out parameter setby the cut-out parameter setting section 114 to cut out the cut-outrange from the omnidirectional image IM31 and performs the distortioncorrection processing for the cut-out range (step S124) to therebygenerate a distortion-corrected image IM32. The masking executionsection 113 inputs the distortion-corrected image IM32 and the shapedmasking area MA33 generated by the masking area shaping section 138. Themasking execution section 113 then performs the masking processing forthe shaped masking area MA33 on the distortion-corrected image IM32(step S125) to thereby generate a distortion-corrected image IM33 onwhich the masking processing is performed for the masking area MA33.

As described above, according to the third embodiment of the presenttechnology, the same parameter is used to perform cut-out/distortioncorrection processing for the masking area and omnidirectional image.The masking area thus corrected is shaped and then applied to theomnidirectional image (distortion-corrected image) that is subjected tothe cut-out/distortion correction processing.

Further, in the shooting system 103 according to the third embodiment,the viewer 230 includes only the output section 213, and therefore theviewer 230 can be realized even in a device with limited processingpower.

Also in the second and third embodiments, as long as the camera 120 or130 includes at least the omnidirectional imaging section 111 and theviewer 220 or 230 includes at least the output section 213, otherconstituent elements may be provided in any one of the camera 120 or 130and viewer 220 or 230. For example, in the shooting system 103 accordingto the third embodiment, the cut-out parameter setting section 114 maybe provided in the viewer 230 as in the shooting system 102 according tothe second embodiment so as to allow the cut-out range to be specifiedon the viewer 230 side.

Further, in the shooting system according to the first to thirdembodiments, some or all of the constituent elements other than theomnidirectional imaging section 111 and output section may be providedin a device provided separately from the camera and viewer andcommunicably located therebetween.

The following describes modifications of shaping of the masking areadescribed in the first to third embodiments.

1-4. Modifications of Shaping of Masking Area 1-4-1. First Modification

FIGS. 13A and 13B are views each explaining a cut-out direction of thecut-out processing performed by the cut-out/distortion correctionsection 115 which is described in the first to third embodiment. FIG.13A illustrates a case where the cut-out range is cut-out in a radialdirection of a circular omnidirectional image, and FIG. 13B illustratesa case where the cut-out range is cut-out in a direction different fromthe radial direction of the circular omnidirectional image (a directionforming an angle α with cut-out direction of FIG. 13A). To cut out thecut-out range in the cut-out/distortion correction processing in a givendirection means that the cut-out processing is performed such that thegiven direction faces upward in a rectangular image obtained by cuttingout the cut-out range and performing distortion correction processingfor the cut-out range.

In a case of FIG. 13A, as a result of generation of adistortion-corrected image by cut-out processing in the radialdirection, an upright direction of a subject including a person M seenin a cut-out range CA11 coincides with a vertical direction in adistortion-corrected image IM42. On the other hand, in a case of FIG.13B, as a result of generation of a distortion-corrected image bycut-out processing in a direction forming an angle α with the radialdirection, an upright direction of a subject including a person M seenin a cut-out range CA12 is inclined by the angle α with respect to thevertical direction in a distortion-corrected image IM43.

FIG. 14 is a view for explaining the shaping of the masking areasdescribed in FIGS. 13A and 13B, respectively. In a case where thecut-out range CA11 is cut out in the radial direction, and where theupright direction of the subject in the distortion-corrected image IM42coincides with the vertical direction of the distortion-corrected imageIM42, a masking area MA41 assuming a distorted shape as a result of thedistortion correction processing is shaped into a masking area MA42having an upright rectangular shape including the masking area MA41,whereby the masking area can be made easy to see for the user.

However, in a case where the cut-out range CA12 is cut out in adirection inclined with respect to the radial direction, and where theupright direction of the subject in the distortion-corrected image IM43is inclined with respect to the vertical direction of thedistortion-corrected image IM43, when a masking area MA43 assuming adistorted shape as a result of the distortion correction processing isshaped into a masking area MA44 having an upright rectangular shapeincluding the masking area MA43, the masking area MA44 is upright,whereas the subject is inclined, which causes strange feeling.

In order to cope with this problem, in the present modification, in thecase where the upright direction of the subject in thedistortion-corrected image IM43 is inclined with respect to the verticaldirection of the distortion-corrected image IM43 due to the inclinationof the cut-out direction, the masking area MA43 assuming a distortedshape as a result of the distortion correction processing is shaped withthe inclination taken into consideration. Specifically, the masking areaMA43 is shaped into a masking area MA45 whose upright direction isinclined at the same angle as the inclination angle of the cut-outdirection with respect to the radial direction.

According to the present modification, in the distortion-corrected imageon which the masking processing is performed for the masking area, theupright direction of the subject and upright direction of the shapedmasking area coincide with each other, whereby a distortion-correctedimage having no strange feeling can be obtained as a result of naturalmasking processing.

1-4-2. Second Modification

FIGS. 15 to 17 are views for explaining a second modification of theshaping of the masking area. First, with reference to FIG. 15,processing of cutting out the cut-out range from the omnidirectionalimage and performing the distortion correction processing for thecut-out range so as to generate a distortion-corrected image will bedescribed. In the present modification, in a case where thecut-out/distortion correction processing is performed for a cut-outrange CA21, pixels in the cut-out range CA21 in an omnidirectional imageIM51 are coordinate-transformed to be projected to thedistortion-corrected image. For example, pixels P11, P12, and P13 in theomnidirectional image IM51 become pixels P21, P22, and P23 on thedistortion-corrected image, respectively. A deficiency of pixels in thedistortion-corrected image is filled in by linearly interpolating thesource coordinates for each square of 16 pixels×16 pixels and, finally,a distortion-corrected image IM52 is generated.

In the present modification, the above squares are used to shape themasking area. That is, the shaping of the masking area is performed on aper square basis. FIG. 16 is a view illustrating a first method of thepresent modification. In this example, when the cut-out/distortioncorrection processing is performed for the cut-out range CA21 on theomnidirectional image IM51 to generate a distortion-corrected imageIM53, a masking area MA51 set in the omnidirectional image IM51 becomesa masking area MA52 having a distorted shape. The masking area 52 isthen shaped into a masking area MA53 including a plurality of squaresthat covers the masking area MA52 and have a rectangular shape as awhole.

Specifically, squares in a row including a square SQ1 including theuppermost point of the masking area MA52 assuming a distorted shape as aresult of the distortion correction processing are set as squaresconstituting the uppermost row of a shaped masking area, squares in acolumn including a square SQ2 including the rightmost point of themasking area MA52 are set as squares constituting the rightmost columnof the shaped masking area, squares in a row including a square SQ3including the lowermost point of the masking area MA52 are set assquares constituting the lowermost row of the shaped masking area, andsquares in a column including a square SQ4 including the leftmost pointof the masking area MA52 are set as squares constituting the leftmostcolumn of the shaped masking area, and a rectangular area surrounded bythe above uppermost row, rightmost column, lowermost row, and leftmostcolumn is set as the shaped masking area MA53.

FIG. 17 is a view illustrating a second method of the presentmodification. In the second method, (only) squares that include themasking area MA52 assuming a distorted shape as a result of thedistortion correction processing are set as a shaped masking area MA54.The shaped masking area MA54 is an area formed by only horizontal andvertical lines. That is, unlike the masking area MA52 before shaping,the masking area MA54 does not include a diagonal line, resulting inimproved appearance.

Although the first to third embodiments of the present technology havebeen described taking a case where the cut-out range is cut out from theomnidirectional image and the distortion correction processing isperformed for the cut-out range, the present technology can be appliedto a case where the distortion correction is performed for theomnidirectional image to generate a panoramic image and then the maskingprocessing is performed for a partial area of the panoramic image.Further, although the omnidirectional image is shot as the image havingdistortion in the above embodiments, the image having distortion is notlimited to the omnidirectional image (having horizontal view angle ofabout 360°). The present technology is effectively applied to a shootingsystem that sets the masking area in the image having distortion andperforms the distortion correction therefor.

2. Fourth to Sixth Embodiments

The following describes fourth to sixth embodiments of the presenttechnology. In any of the fourth to sixth embodiments, a shooting systemin which a camera using a fish-eye lens is used to image theomnidirectional image as the image having distortion is employed.However, the image having distortion in the present technology is notlimited to the omnidirectional image but may be an image havingdistortion due to imaging using a wide-angle lens.

2-1. Fourth Embodiment

FIG. 20 is a block diagram illustrating an image display systemaccording to an embodiment of the present technology. As illustrated inFIG. 20, a shooting system 104 includes a camera 140 and a viewer 240connected to each other. In a case where the camera 140 is a monitoringcamera, the camera 140 is installed so as to be able to image a place tobe monitored, and the viewer 240 is a device for a user to view an imageimaged by the camera 140 and subjected to image processing. The viewer240 may be, e.g., a personal computer or a mobile terminal. The camera140 is so-called a network camera and may be connected to the viewer 240by wire or wireless. Further, some or all of the constituent elementsother than an omnidirectional imaging section 141 of the camera 140 maybe provided in the viewer 240.

The camera 140 includes an omnidirectional imaging section 141, amasking area reception section 142, a masking frame generation section143, a masking frame superimposing section 144, a masking executionsection 145, a cut-out/distortion correction section 146, and asynthesis section 147. The viewer 240 includes an input section 241, amasking area determination section 242, a display switching section 243,and a display section 244.

The omnidirectional imaging section 141 of the camera 140 is an imagingsystem having a fish-eye lens which is a kind of a wide-angle lens. Theomnidirectional imaging section 141 is used to image a circularomnidirectional image. The omnidirectional image is the image havingdistortion in which a degree of the distortion increases with proximityto an outer periphery thereof. The omnidirectional imaging section 141outputs the omnidirectional image obtained by imaging to the maskingframe superimposing section 144 and masking execution section 145, aswell as, to the viewer 240. The image taken and output by theomnidirectional imaging section 141 may be a still image or a movingimage. In the latter case, the following processing is performed foreach frame of the moving image.

The input section 241 of the viewer 240 inputs various instructions fromthe user. The instruction includes a display switching instruction, amasking area specification instruction, a masking areacancel/determination instruction, and the like. The display switchinginstruction is input to the display switching section 243, and themasking area specification instruction and masking areacancel/determination instruction are input to the masking areadetermination section 242. The input section 241 includes, e.g., a mouseand a keyboard or a touch sensor formed integrally with the displaysection 244.

The masking area determination section 242 calculates the masking areafrom input data input to the input section 241. Specifically, arectangle (oblong) formed by horizontal and vertical sides extendingfrom a vertex on the omnidirectional image clicked by a mouse and acurrently-pointed vertex at a diagonal position with respect to thevertex is calculated. Information concerning a position and a shape(length in a vertical direction and a width in a horizontal direction)of the omnidirectional image is transmitted to the camera 140 asinformation specifying the masking area.

The masking area reception section 142 receives the informationspecifying the masking area determined by the masking area determinationsection 242 and output the received information to the masking framegeneration section 143 and masking execution section 145. The maskingframe generation section 143 generates a masking frame representing anouter edge of the masking area based on the masking area specificationinformation acquired from the masking area reception section 142. Themasking frame includes information indicating a position of the maskingframe in the omnidirectional image.

The masking frame superimposing section 144 acquires the omnidirectionalimage from the omnidirectional imaging section 141, as well as, acquiresthe masking frame from the masking frame generation section 143 andsuperimposes a shape of the masking frame on the omnidirectional image.The masking frame superimposing section 144 then outputs theomnidirectional image on which the masking frame is superimposed to thesynthesis section 147 and display switching section 243.

The masking execution section 145 acquires the omnidirectional imagefrom the omnidirectional imaging section 141, as well as, acquires themasking area specification information from the masking area receptionsection 142 and then performs the masking processing for the maskingarea included in the omnidirectional image and corresponding to thespecification information. The masking processing performed by themasking execution section 145 is processing of blacking out the maskingarea. However, the masking processing is not limited to this, but may beanother processing, such as mosaic processing, that reduces aninformation amount of an image corresponding to the masking area. Themasking execution section 145 outputs, to the cut-out/distortioncorrection section 146, the omnidirectional image on which the maskingprocessing is performed for the masking area.

The cut-out/distortion correction section 146 cuts out the cut-out rangefrom the omnidirectional image on which the masking processing isperformed for the masking area and performs the distortion correctionfor the cut-out range to thereby generate a cut-out/distortion-correctedimage. In the present embodiment, the cut-out range is previouslydetermined. A frame representing the cut-out range may be displayed in asuperimposed manner on the omnidirectional image. The cut-out/distortioncorrection section 146 outputs the cut-out/distortion-corrected image tothe synthesis section 147, as well as transmits the same to the viewer240.

The synthesis section 147 acquires the omnidirectional image on whichthe masking frame is superimposed from the masking frame superimposingsection 144 and acquires the cut-out/distortion-corrected image on whichthe masking processing is already performed for the masking area fromthe cut-out/distortion correction section 146 and then synthesizes themto thereby generate a preview image displaying the both imagessimultaneously. The synthesis section 147 transmits the generatedpreview image to the viewer 240.

The display switching section 243 receives the omnidirectional imagefrom the omnidirectional imaging section 141, receives theomnidirectional image on which the masking frame is superimposed fromthe masking frame superimposing section 144, receives thecut-out/distortion-corrected image on which the masking processing isperformed for the masking area from the cut-out/distortion correctionsection 146, and receives the preview image from the synthesis section147.

Hereinafter, a display mode of the shooting system 104 will bedescribed. The shooting system 104 switches between a masking areaspecifying mode for determining the masking area and a normal mode fordisplaying the cut-out/distortion-corrected image on which the maskingprocessing is performed for the masking area. These display modes areswitched based on an input from the input section 241. The displayswitching section 243 switches the display mode by acquiring modeswitching information from the input section 241.

In the masking area specifying mode, the display switching section 243outputs first the omnidirectional image acquired from theomnidirectional imaging section 141 to the display section 244. Whenacquiring the omnidirectional image on which the masking frame issuperimposed from the masking frame superimposing section 144 in themasking area specifying mode, the display switching section 243 outputsthe omnidirectional image on which the masking frame is superimposed tothe display section 244. When acquiring the preview image from thesynthesis section 147 in the masking area specifying mode, the displayswitching section 243 outputs the preview image to the display section244. Further, in the normal mode, the display switching section 243outputs, to the display section 244, one of the omnidirectional imageacquired from the omnidirectional imaging section 141 andcut-out/distortion-corrected image on which the masking processing isperformed for the masking area acquired from the cut-out/distortioncorrection section 146. Which one of the omnidirectional image andcut-out/distortion-corrected image on which the masking processing isperformed for the masking area is output is determined according to aninstruction from the input section 241.

The display section 244 displays the image acquired from the displayswitching section 243. The display section 244 is, e.g., a liquidcrystal display device provided with a liquid crystal display panel.When the input section 241 is configured as a touch sensor, the displaysection 244 is configured as a touch panel formed integrally with thetouch sensor.

FIG. 19 and FIGS. 21 to 24 each illustrate a display example on thedisplay section 244. FIG. 21 is a view illustrating an example of theomnidirectional image according to the present embodiment. A horizontalview angle of the omnidirectional imaging section 141 is 360° and, thus,the omnidirectional image has a circular shape as illustrated in FIG.21. The circular omnidirectional image includes distortion, and a degreeof the distortion is not uniform over the entire image area butincreases with proximity to an outer periphery of the circle. Theomnidirectional image illustrated in FIG. 21 is displayed when displayof the omnidirectional image is selected in the normal mode or when themasking area is specified in the masking area specifying mode.

FIG. 22 is a view illustrating an example of the masking areaspecification according to the embodiment of the present technology. Inthe present embodiment, the masking area is specified on theomnidirectional image. As illustrated in FIG. 22, a pointer movingaccording to operation of a mouse as the input section 241 is displayedon the omnidirectional image, and a start point of the masking area isspecified by mouse click. When the pointer is moved after the startpoint is specified, the masking area determination section 242calculates a rectangle formed by two sides parallel to the verticaldirection of a screen and two sides parallel to the horizontal directionof the screen and having diagonal points of the start point and apointer position. The masking frame superimposing section 144 usesinformation of the rectangle (specification information of the maskingarea) to generate the omnidirectional image on which the masking frameis superimposed, and the display section 244 displays theomnidirectional image.

FIG. 23 is a view illustrating an example of the masking areaspecification according to the embodiment of the present technology. Aframe of the calculated rectangular is displayed in a superimposedmanner on the omnidirectional image. When the pointer is moved to form aframe representing a desired masking area, the mouse is clicked onceagain at this position. Then, the preview image is generated by usingthe masking area specification information at this time and displayed onthe display section 244.

FIG. 19 is a view illustrating an example of the preview image accordingto the embodiment of the present technology. In the preview image, thecut-out/distortion-corrected image on which the masking processing isperformed for the masking area is displayed in a superimposed manner onthe omnidirectional image on which the masking frame is superimposed. Atthis time, the cut-out/distortion-corrected image is disposed in aballoon popped up from the masking frame superimposed on theomnidirectional image.

A “cancel” button and a “determination” button are displayed togetherwith the preview image. A user confirms the cut-out/distortion-correctedimage in the balloon. When the user is dissatisfied with the maskingarea, the user can cancel this masking area by clicking the “cancel”button. When the masking area is canceled, the display switching section243 switches the display mode to the masking area specifying modeillustrated in FIG. 22. The user confirms thecut-out/distortion-corrected image in the balloon. When the user issatisfied with the masking area in the cut-out/distortion-correctedimage, the user clicks the “determination” button to decide this maskingarea.

FIG. 24 is a view illustrating the cut-out/distortion-corrected imageaccording to the embodiment of the present technology. Thecut-out/distortion-corrected image is displayed when display of thecut-out/distortion-corrected image is selected in the normal mode. Inthe cut-out/distortion-corrected image, the masking processing isperformed for the decided masking area. However, this masking area isspecified and decided as a rectangular area on the omnidirectionalimage, so that the masking area on this cut-out/distortion-correctedimage assumes a distorted shape as a result of the distortion correctionperformed when the cut-out/distortion-corrected image is generated fromthe omnidirectional image.

FIG. 25 is a flow diagram for explaining a procedure of generating thepreview image in the fourth embodiment of the present technology. Withreference to FIG. 25, a procedure of generating the preview image in thepresent embodiment will be described. The omnidirectional imagingsection 141 images an omnidirectional image D71. Then, a masking areaD72 is specified on the omnidirectional image. The masking framesuperimposing section 144 superimposes a masking frame generated usingthe masking area D72 on the omnidirectional image D71 to generate anomnidirectional image D73 in which the masking frame is superimposed.The masking execution section 145 performs the masking processing forthe masking area D72 on the omnidirectional image D71 to generate anomnidirectional image D74 on which the masking processing is performedfor the masking area.

Then, the cut-out/distortion correction section 146 cuts out the cut-outrange from the omnidirectional image D74 on which the masking processingis performed for the masking area and performs the distortion correctionprocessing for the cut-out range to thereby generate acut-out/distortion-corrected image D75 on which the masking processingis performed for the masking area. Finally, the synthesis section 147superimposes a balloon including therein thecut-out/distortion-corrected image D75 on which the masking processingis performed for the masking area on the omnidirectional image D73 inwhich the masking frame is superimposed on the masking area to therebygenerate a preview image D76.

FIG. 26 is an operational flow diagram of the shooting system accordingto the fourth embodiment of the present technology. The flow of FIG. 26illustrates operation of the shooting system 104 in the masking areaspecifying mode. With reference to FIG. 26, operation of the shootingsystem 104 according to the fourth embodiment will be described.

The omnidirectional imaging section 141 images an omnidirectional image,and the display section 244 displays the omnidirectional image (stepS261). Then, it is determined whether or not the masking areaspecification is started (step S262) by the input section 241.Specifically, it is determined that the masking area specification isstarted by detecting that a start point of the masking area is clickedby operation of the input section 241. When the masking areaspecification is not started (NO in step S262), operation stands byuntil start of the masking area specification while displaying theomnidirectional image.

When the masking area specification is started, that is, when the startpoint of the masking area is clicked on the omnidirectional image (YESin step S262), the masking area determination section 242 calculates themasking area according to movement of the mouse pointer and transmitsthe specification information of the calculated masking area to thecamera 140, and the masking area reception section 142 acquires themasking area specification information (step S263). Then, the maskingframe generation section 143 generates the masking frame based on themasking area specification information (step S264), and the maskingframe superimposing section 144 superimposes the masking frame on theomnidirectional image (step S265). The generation of the masking frameand superimposition thereof on the omnidirectional image are performedaccording to a position of the pointer while the pointer is being movedafter start point is clicked.

Then, when an end point of the masking area is clicked, the masking areais temporarily decided, and the masking execution section 145 performsthe masking processing for the temporarily decided masking area(hereinafter, referred to as simply “masking area”) on theomnidirectional image (step S266). Then, the cut-out/distortioncorrection section 146 cuts out the cut-out range from theomnidirectional image on which the masking processing is performed forthe masking area and performs the distortion correction processing forthe cut-out range to thereby generate the cut-out/distortion-correctedimage (step S267). In the generated cut-out/distortion-corrected image,the masking processing is performed for the masking area. The synthesissection 147 synthesizes the omnidirectional image on which the maskingframe is superimposed and cut-out/distortion-corrected image on whichthe masking processing is performed for the masking area to therebygenerate the preview image (step S268) and transmits the generatedpreview image to the viewer 240.

When receiving the preview image, the display switching section 243 ofthe viewer 240 outputs the preview image to the display section 244, andthe display section 244 displays the preview image (step S269). It isdetermined whether or not the user decides the masking area displayed inthe preview image through the input section 241 (step S270). When themasking area displayed in the preview image is decided (YES in stepS270), this processing terminates. When the user cancels the maskingarea displayed in the preview image through the input section 241 (NO instep S270), the display switching section 243 switches the image to beoutput to the display section 244 to the omnidirectional image, and theflow returns to step S263 of acquiring the masking area specificationinformation again.

As described above, according to the shooting system 104 of the presentembodiment and an image display method executed thereby, the maskingarea is specified on the omnidirectional image, and the specifiedmasking area is displayed in the preview image so as to be recognizableon the omnidirectional image. In the preview image, thecut-out/distortion-corrected image on which the entire image areaincluding the masking area is subjected to the distortion correction andin which the masking area is included in a recognizable manner isdisplayed simultaneously. Thus, it can be confirmed what position andshape the masking area specified on the omnidirectional image assumes.When the user is satisfied with the displayed masking area as a resultof the conformation, he or she instructs the “determination” for thepreview image to decide the masking area.

2-2. Fifth Embodiment

The following describes a fifth embodiment of the present technology.FIG. 27 is a block diagram of a shooting system according to the fifthembodiment of the present technology. As illustrated in FIG. 27, ashooting system 105 includes a camera 150 and a viewer 250 connected toeach other. In a case where the camera 150 is a monitoring camera, thecamera 150 is installed so as to be able to image a place to bemonitored, and the viewer 250 is a device for a user to view an imagetaken by the camera 140 and subjected to image processing. The viewer250 may be, e.g., a personal computer or a mobile terminal. The camera150 is so-called a network camera and may be connected to the viewer 250by wire or wireless. Further, some or all of the constituent elementsother than an omnidirectional imaging section 151 of the camera 150 maybe provided in the viewer 250.

The camera 150 includes an omnidirectional imaging section 151, amasking area reception section 152, a cut-out range determinationsection 153, a masking execution section 154, cut-out/distortioncorrection section 155, and a synthesis section 156. The viewer 250includes an input section 251, a masking area determination section 252,a masking frame generation section 253, a masking frame superimposingsection 254, a display switching section 255, and a display section 256.

Hereinafter, descriptions of the same configurations as those of thefourth embodiment will be omitted. The present embodiment differs fromthe fourth embodiment mainly in the following two points. The firstdifference is a difference in the procedure of generating the previewimage and distribution of roles between the camera 150 and viewer 250for the preview image generation. That is, in the fourth embodiment, inthe camera 140, the masking execution section 145 performs the maskingprocessing on the omnidirectional image, and then the cut-out/distortioncorrection section 146 generates the cut-out/distortion-corrected imageon which the masking processing is performed for the masking area, whilethe masking frame superimposing section 144 superimposes the frame ofthe masking area on the omnidirectional image to generate theomnidirectional image on which the masking frame is superimposed, andthe synthesis section 147 synthesizes the cut-out/distortion-correctedimage on which the masking processing is performed for the masking areaand omnidirectional image on which the masking frame is superimposed togenerate the preview image illustrated in FIG. 19. On the other hand, inthe present embodiment, in the camera 150, the masking processing isperformed on the omnidirectional image, followed by thecut-out/distortion correction processing, to generate thecut-out/distortion-corrected image, as in the case of the fourthembodiment; however, the generated cut-out/distortion-corrected image isfirst synthesized with the omnidirectional image, and resultantsynthetic image is transmitted to the viewer 250. The viewer 250superimposes the masking frame on the synthetic image to therebygenerate the preview image as illustrated in FIG. 19.

The second difference is that while the cut-out range in thecut-out/distortion correction section 146 is fixed in the fourthembodiment, the cut-out range is determined based on the masking areaspecification information in the present embodiment. A more specificdescription will be made below on this point.

When receiving input data from the input section 251, the masking areadetermination section 252 calculates the masking area and outputs thespecification information representing the masking area to the maskingframe generation section 253, as well as, to the camera 150. The maskingarea reception section 152 acquires the masking area specificationinformation and outputs the acquired masking area specificationinformation to the cut-out range determination section 153 and maskingexecution section 154. The cut-out range determination section 153determines the cut-out range based on the masking area specificationinformation acquired from the masking area reception section 152. Thecut-out range determination section 153 stores a relationship betweenthe masking area represented by the specification information andcut-out range. When receiving the masking area specificationinformation, the cut-out range determination section 153 calculates thecut-out range corresponding to the masking area specificationinformation.

Various relationships between the masking area represented by thespecification information and cut-out range can be considered. In thepresent embodiment, the cut-out range determination section 153determines, as the cut-out range, a rectangular area having the samecenter and orientation as those of the masking area and having a size10/7 that the masking area.

The masking execution section 154 performs, on the omnidirectional imageacquired from the omnidirectional imaging section 151, the maskingprocessing for the masking area acquired from the masking area receptionsection 152. The cut-out/distortion correction section 155 cuts out thecut-out range determined by the cut-out range determination section 153from the omnidirectional image on which the masking processing isperformed for the masking area to thereby generate thecut-out/distortion-corrected image on which the masking processing isperformed for the masking area. The generatedcut-out/distortion-corrected image is output to the synthesis section156.

In the present embodiment, the omnidirectional image taken by theomnidirectional imaging section 151 is directly output to the synthesissection 156 and transmitted to the viewer 250. The synthesis section 156synthesizes the omnidirectional image and cut-out/distortion-correctedimage generated by the cut-out/distortion correction section 155 tothereby generate a synthetic image. As illustrated in FIG. 19, in thesynthetic image, the balloon including the cut-out/distortion-correctedimage is superimposed on the omnidirectional image. However, in thissynthetic image, the masking frame representing the masking areaspecified by the user is not superimposed on the omnidirectional image.The synthesis section 156 transmits the generated synthetic image to theviewer 250.

The masking frame superimposing section 254 of the viewer 250 receivesthe synthetic image. On the other hand, the masking frame generationsection 253 generates a masking frame based on the masking areaspecification information acquired from the masking area determinationsection 252 and outputs the generated masking frame to the masking framesuperimposing section 254. The masking frame superimposing section 254superimposes the masking frame acquired from the masking framegeneration section 253 on the synthetic image acquired from the camera150 to thereby generate the preview image as illustrated in FIG. 19 andoutputs the preview image to the display switching section 255.

When directly receiving the omnidirectional image from theomnidirectional imaging section 151, the masking frame superimposingsection 254 superimposes the masking frame generated by the maskingframe generation section 253 on the omnidirectional image and outputsthe resultant image to the display switching section 255. The maskingframe superimposing section 254 superimposes, according to a pointerposition, a rectangular masking frame defined by a start point of themasking area and an endpoint corresponding to the pointer positionduring a time between specification of the start point of the maskingarea and end point thereof on the omnidirectional image. At this time,the masking frame superimposing section 254 superimposes the maskingframe on the omnidirectional image acquired from the omnidirectionalimaging section 151. After the masking area is temporarily decided, thecut-out/distortion correction section 155 generates thecut-out/distortion-corrected image on which the masking processing isperformed for the temporality decided masking area, and then thesynthesis section 156 generates the synthetic image on which thecut-out/distortion-corrected image is included in the balloon andtransmits the synthetic image to the viewer 250. When receiving thesynthetic image, the masking frame superimposing section 254superimposes the masking frame not on the omnidirectional image acquiredfrom the omnidirectional imaging section 151 but on the receivedsynthetic image to generate the preview image and outputs the previewimage to the display switching section 255.

The display switching section 255 acquires the omnidirectional imagefrom the omnidirectional imaging section 151, acquires the preview imageor omnidirectional image on which the masking frame is superimposed fromthe masking frame superimposing section 254, and acquires thecut-out/distortion-corrected image on which the masking processing isperformed for the masking area from the cut-out/distortion correctionsection 155. The display switching section 255 acquires display modeswitching information from the input section 251 and performs switchingof the display mode.

In the masking area specifying mode, the display switching section 255outputs the omnidirectional image acquired from the omnidirectionalimaging section 151 to the display section 256. When acquiring theomnidirectional image on which the masking frame is superimposed fromthe masking frame superimposing section 254 in the masking areaspecifying mode, the display switching section 255 outputs theomnidirectional image on which the masking frame is superimposed to thedisplay section 256, while acquiring the preview image from the maskingframe superimposing section 254, the display switching section 255outputs the preview image to the display section 256. In the normalmode, the display switching section 255 outputs, to the display section256, one of the omnidirectional image acquired from the omnidirectionalimaging section 151 and cut-out/distortion-corrected image on which themasking processing is performed for the masking area acquired from thecut-out/distortion correction section 155. Which one of theomnidirectional image and cut-out/distortion-corrected image on whichthe masking processing is performed for the masking area is to be outputis determined according to an instruction from the input section 251.

FIG. 28 is a flow diagram for explaining a procedure of generating thepreview image in the fifth embodiment of the present technology. Withreference to FIG. 28, a procedure of generating the preview image in thepresent embodiment will be described. The omnidirectional imagingsection 151 images an omnidirectional image D11. Then, a masking areaD12 is specified on the omnidirectional image. The masking executionsection 154 performs the masking processing for the masking area D12 onthe omnidirectional image D11 to thereby generate an omnidirectionalimage D13 on which the masking processing is performed for the maskingarea. The cut-out range determination section 153 determines the cut-outrange based on the masking area D12. The cut-out/distortion correctionsection 155 cuts out the cut-out range from the omnidirectional imageD13 on which the masking processing is performed for the masking areaand performs the distortion correction processing for the cut-out rangeto thereby generate a cut-out/distortion-corrected image D14. Thesynthesis section 156 superimposes a balloon including therein thecut-out/distortion-corrected image D14 on which the masking processingis performed for the masking area on the omnidirectional image D11 tothereby generate a synthetic image D15.

In the viewer 250, the masking frame generation section 253 generates amasking frame D16. The masking frame superimposing section 254superimposes the masking frame D16 on the synthetic image D15 to therebygenerate a preview image D17.

FIG. 29 is an operational flow diagram of the shooting system accordingto the fifth embodiment of the present technology. The flow of FIG. 29illustrates operation of the shooting system 105 in the masking areaspecifying mode. With reference to FIG. 29, an image display methodaccording to the fifth embodiment will be described.

The omnidirectional imaging section 151 images an omnidirectional image,and the display section 256 displays the omnidirectional image (stepS291). Then, it is determined whether or not the masking areaspecification is started by the input section 251 (step S292).Specifically, it is determined that the masking area specification isstarted by detecting that a start point of the masking area is clickedby operation of the input section 251. When the masking areaspecification is not started (NO in step S292), operation stands byuntil start of the masking area specification while displaying theomnidirectional image.

When the masking area specification is started, that is, when the startpoint of the masking area is clicked on the omnidirectional image (YESin step S292), the masking area determination section 252 calculates themasking area according to movement of the mouse pointer and transmitsthe specification information of the calculated masking area to thecamera 150, as well as to the masking frame generation section 253. Themasking area reception section 152 of the camera 150 acquires themasking area specification information (step S293). Further, the maskingframe generation section 253 of the viewer 250 generates the maskingframe based on the masking area specification information (step S294)and outputs the generated masking frame to the masking framesuperimposing section 254. The masking frame superimposing section 254superimposes the masking frame on the omnidirectional image, and thedisplay section 256 displays the resultant image. The generation of themasking frame and superimposition thereof on the omnidirectional imageare performed according to a position of the pointer while the pointeris being moved after start point is clicked.

Then, when an end point of the masking area is clicked, the masking areais temporarily decided, and the masking execution section 154 performsthe masking processing for the temporarily decided masking area on theomnidirectional image (step S295), and cut-out range determinationsection 153 determines the cut-out range based on the temporarilydecided masking area (step S296). Then, the cut-out/distortioncorrection section 155 cuts out the cut-out range from theomnidirectional image on which the masking processing is performed forthe masking area and performs the distortion correction processing forthe cut-out range to thereby generate the cut-out/distortion-correctedimage (step S297). In the generated cut-out/distortion-corrected image,the masking processing is performed for the masking area. The synthesissection 156 synthesizes the omnidirectional image acquired from theomnidirectional imaging section 151 and cut-out/distortion-correctedimage on which the masking processing is performed for the masking areato thereby generate the synthetic image (step S298) and transmits thegenerated synthetic image to the viewer 250.

When receiving the synthetic image, the masking frame superimposingsection 254 superimposes the masking frame generated by the maskingframe generation section 253 on the synthetic image to thereby generatethe preview image and outputs the generated preview image to the displayswitching section 255 (step S299), and the display section 256 displaysthe preview image (step S300). It is determined whether or not the userdecides the masking area displayed in the preview image through theinput section 251 (step S301). When the masking area displayed in thepreview image is decided (YES in step S301), this processing terminates.When the user cancels the masking area displayed in the preview imagethrough the input section 251 (NO in step S301), the display switchingsection 253 switches the image to be output to the display section 254to the omnidirectional image, and the flow returns to step S293 ofacquiring the masking area specification information again.

As described above, in the present embodiment, the masking frame isgenerated in the viewer 250 and, further, in the viewer 250, the previewimage is generated by superimposing the masking frame on the synthesisimage or the omnidirectional image on which the masking frame issuperimposed is generated. As described above, for specifying themasking area, the start point is specified, and then a rectangular framehaving two diagonal vertexes of the start point and a current pointerposition is generated and superimposed on the omnidirectional image. Atthis time, when a configuration is such that information of the startpoint and pointer position is transmitted to the camera 150, therectangular frame is generated and superimposed on the omnidirectionalimage on the camera 150 side, and the resultant image is transmitted tothe viewer 250, it may be difficult to display the rectangular frame onthe omnidirectional image following movement of the pointer when acommunication situation is bad. On the other hand, in the presentembodiment, the masking frame is generated in the viewer 250 providedwith the input section 251 for specifying the pointer position anddisplay section 256 for displaying the omnidirectional image on whichthe masking frame is superimposed and superimposed on theomnidirectional image. Thus, irrespective of the communication situationbetween the camera 150 and viewer 250, the masking frame can bedisplayed in a superimposed manner on the omnidirectional image withoutdelay.

Further, in the present embodiment, the cut-out range determinationsection 153 determines the cut-out range based on the masking areaspecification area, so that it is possible to confirm thecut-out/corrected image including the masking area together with theomnidirectional image. The configuration for determining the cut-outrange based on the masking area specification information which has beendescribed in the present embodiment can be applied to the shootingsystem according to the other embodiments of the present technology.

2-3. Sixth Embodiment

The following describes a sixth embodiment of the present technology.FIG. 30 is a block diagram of an image display system according to thesixth embodiment of the present technology. As illustrated in FIG. 30, ashooting system 106 includes a camera 160 and a viewer 260 connected toeach other. In a case where the camera 160 is a monitoring camera, thecamera 160 is installed so as to be able to image a place to bemonitored, and the viewer 260 is a device for a user to view an imageimaged by the camera 160 and subjected to image processing. The viewer260 may be, e.g., a personal computer or a mobile terminal. The camera160 is so-called a network camera and may be connected to the viewer 260by wire or wireless. Further, some or all of the constituent elementsother than an omnidirectional imaging section 161 of the camera 160 maybe provided in the viewer 260.

The camera 160 includes an omnidirectional imaging section 161, amasking area reception section 162, a masking frame generation section163, a masking frame superimposing section 164, a masking areadistortion correction section 165, a cut-out masking execution section166, a cut-out correction section 167, and a synthesis section 168. Theviewer 260 includes an input section 261, a masking area determinationsection 262, a display switching section 263, and a display section 264.

Hereinafter, descriptions of the same configurations as those of thefourth embodiment will be omitted. In the present embodiment, forgeneration of the cut-out/distortion-corrected image on which themasking processing is performed for the masking area, cut-out/distortioncorrection processing is performed for the omnidirectional image togenerate the cut-out/distortion-corrected image, while thecut-out/distortion correction processing is performed also for themasking area using the same correction parameter (transform matrix) asthat used in the cut-out/distortion correction processing for theomnidirectional image to thereby generate a cut-out/distortion-correctedmasking area. Then, the masking processing is performed for thecut-out/distortion-corrected masking area in thecut-out/distortion-corrected image to generate thecut-out/distortion-corrected image on which the masking processing isperformed for the masking area. A more specific description will be madebelow on this point.

When the input section 261 inputs coordinates of the start point andendpoint specified on the omnidirectional image, the masking areadetermination section 262 calculates the masking area and transmits thecorresponding masking area specification information to the camera 160.The masking area reception section 162 of the camera 160 receives themasking area specification information and outputs the received maskingarea specification information to the masking area distortion correctionsection 165 and masking frame generation section 163.

The masking frame generation section 163 generates the masking framebased on the masking area specification information acquired from themasking area reception section 162 and outputs the generated maskingframe to the masking frame superimposing section 164. The masking framesuperimposing section 164 acquires the omnidirectional image from theomnidirectional imaging section 161 and superimposes the masking frameon the omnidirectional image to thereby generate the omnidirectionalimage on which the masking frame is superimposed and outputs theresultant omnidirectional image to the synthesis section 168.

On the other hand, the masking area distortion correction section 165performs the cut-out/distortion correction processing for the maskingarea based on the masking area specification information acquired fromthe masking area reception section 162 to thereby generate thecut-out/distortion-corrected masking area and outputs the generatedcut-out/distortion-corrected masking area to the cut-out maskingexecution section 166. The cut-out correction section 167 performs thecut-out/distortion correction processing for the omnidirectional imageacquired from the omnidirectional imaging section 161 to therebygenerate the cut-out/distortion-corrected image and outputs thegenerated cut-out/distortion-corrected image to the cut-out maskingexecution section 166.

The cut-out masking execution section 166 performs the maskingprocessing for the cut-out/distortion-corrected masking area on thecut-out/distortion-corrected image to thereby generate thecut-out/distortion-corrected image on which the masking processing isperformed for the masking area and outputs the generatedcut-out/distortion-corrected image to the synthesis section 168. Thesynthesis section 168 acquires the omnidirectional image on which themasking frame is superimposed from the masking frame superimposingsection 164, acquires the cut-out/distortion-corrected image on whichthe masking processing is performed for the masking area from thecut-out masking execution section 166, and uses the acquired images togenerate the preview image as illustrated in FIG. 19. The displayswitching section 263 and display section 264 in the viewer 260 are thesame as the display switching section 243 and display section 244 of thefourth embodiment, so descriptions thereof will be omitted.

FIG. 31 is a flow diagram for explaining a procedure of generating thepreview image in the sixth embodiment of the present technology. Withreference to FIG. 31, a procedure of generating the preview image in thepresent embodiment will be described. The omnidirectional imagingsection 161 images an omnidirectional image D31. Then, a masking areaD32 is specified on the omnidirectional image D3. Then, the maskingframe generation section 163 generates the masking frame, and themasking frame superimposing section 164 superimposes the masking frameon the omnidirectional image D31 to thereby generate an omnidirectionalimage D34 on which the masking frame is superimposed.

On the other hand, the cut-out correction section 167 cuts out thecut-out range from the omnidirectional image D31 and performs thedistortion correction processing for the cut-out range to therebygenerate a cut-out/distortion-corrected image D35. The masking areadistortion correction section 165 performs the same cut-out/distortioncorrection processing as that performed by the cut-out correctionsection 167 for a masking area D32 to thereby generate acut-out/distortion-corrected masking area D36. Then, the cut-out maskingexecution section 166 performs the masking processing for thecut-out/distortion-corrected masking area D36 on thecut-out/distortion-corrected image D35 to thereby generate acut-out/distortion-corrected image D37 on which the masking processingis performed for the masking area. Finally, the synthesis section 168superimposes, on the omnidirectional image D34 in which the maskingframe is superimposed on the masking area, a balloon including thereinthe cut-out/distortion-corrected image D37 on which the maskingprocessing is performed for the masking area to thereby generate apreview image D38.

FIG. 32 is an operational flow diagram of the shooting system accordingto the sixth embodiment of the present technology. The flow of FIG. 32illustrates operation of the shooting system 106 in the masking areaspecifying mode. With reference to FIG. 32, operation of an imagedisplay method according to the sixth embodiment will be described.

The omnidirectional imaging section 161 images an omnidirectional image,and the display section 264 displays the omnidirectional image (stepS321). Then, it is determined whether or not the masking areaspecification is started (step S322) by the input section 261.Specifically, it is determined that the masking area specification isstarted by detecting that a start point of the masking area is clickedby operation of the input section 261. When the masking areaspecification is not started (NO in step S322), operation stands byuntil start of the masking area specification while displaying theomnidirectional image.

When the masking area specification is started, that is, when the startpoint of the masking area is clicked on the omnidirectional image (YESin step S322), the masking area determination section 262 calculates themasking area according to movement of the mouse pointer and transmitsthe specification information of the calculated masking area to thecamera 160. When the masking area reception section 162 of the camera160 acquires the masking area specification information (step S323), themasking frame generation section 163 generates the masking frame basedon the masking area specification information (step S324), the maskingframe superimposing section 164 superimposes the masking frame on theomnidirectional image (step S325), and the display section 264 displaysthe resultant image. The generation of the masking frame andsuperimposition thereof on the omnidirectional image are performedaccording to a position of the pointer while the pointer is being movedafter start point is clicked.

Then, when an end point of the masking area is clicked, the masking areais temporarily decided, and the masking frame superimposing section 164outputs the omnidirectional image on which the masking frame issuperimposed to the synthesis section 168. The cut-out correctionsection 167 performs the cut-out/distortion correction processing forthe omnidirectional image to thereby generate thecut-out/distortion-corrected image (step S326). The masking areadistortion correction section 165 performs the same correctionprocessing as the cut-out/distortion correction processing performed instep S326 for a masking area to thereby generate thecut-out/distortion-corrected masking area (step S327). Then, the cut-outmasking execution section 166 performs the masking processing for thecut-out/distortion-corrected masking area, generated in step S327, onthe cut-out/distortion-corrected image, generated in step S326, on whichthe masking processing is performed (step S328).

The synthesis section 168 acquires the omnidirectional image on whichthe masking frame is superimposed from the masking frame superimposingsection 164 and acquires the cut-out/distortion-corrected image on whichthe masking processing is performed for the masking area from thecut-out masking execution section 166. Then, the synthesis section 168synthesizes the acquired images to generate the preview image asillustrated in FIG. 19 (step S329) and outputs the generated previewimage to the display switching section 263. Then, the display section264 displays the preview image (step S330). It is determined whether ornot the user decides the masking area displayed in the preview imagethrough the input section 261 (step S331). When the masking areadisplayed in the preview image is decided (YES in step S331), thisprocessing terminates. When the user cancels the masking area displayedin the preview image through the input section 261 (NO in step S331),the display switching section 263 switches the image to be output to thedisplay section 264 to the omnidirectional image, and the flow returnsto step S323 of acquiring the masking area specification informationagain.

As described above, in the present embodiment, the cut-out/distortioncorrection processing is performed for the masking area andomnidirectional image using the same correction parameter, and thedistortion-corrected image is subjected to the masking processing withthe masking area obtained by the correction processing. According to thepresent embodiment, the masking processing is executed after thedistortion correction processing, so that it is possible to use adisplay method of, e.g., not displaying a masked image but depictingonly the frame in the preview image. Further, as described above, theconfiguration for determining the cut-out range based on the maskingarea specification information which has been described in the secondembodiment can be applied to the present embodiment. In this case,information of the cut-out range determined by the masking areaspecification information is shared between the cut-out correctionsection 167 and masking area distortion correction section 165.

The fourth to sixth embodiments have thus been described. Variousmodifications can be made to the fourth to sixth embodiments. Thefollowing describes the various modifications.

2-4. Modifications

FIGS. 33A to 33D are views each illustrating a modification ofinstruction operation of the preview image display of the presenttechnology. FIGS. 33A to 33D illustrate display examples of operationfrom specification of the masking area to instruction of the previewimage display in a case where a touch panel including a liquid crystalpanel serving as the display section and a touch sensor serving as theinput section is used. FIG. 33A is a view illustrating an example inwhich the masking area is specified on the omnidirectional image. Theuser makes a pinch-in operation on the touch panel displaying theomnidirectional image to thereby specify the masking area. Here, as inthe case of the above embodiments, a rectangular area formed by verticaland horizontal sides having two points specified by the pinch-inoperation as the vertexes is set as the masking area.

FIG. 33B is a view illustrating an example in which the cut-out range isspecified on the omnidirectional image. The user makes a pinch-outoperation on the omnidirectional image on which the masking area issuperimposed to thereby specify the cut-out range. Here, a shape inwhich two points specified by the pinch-out operation is two diagonalvertexes in the cut-out/corrected image whose cut-out direction is theradial direction is set as the cut-out range. FIG. 33C is a viewillustrating an example in which an instruction of display of thepreview image on the omnidirectional image is issued. When the userflicks the masking area or cut-out range, thecut-out/distortion-corrected image on which the masking processing isperformed for the masking area on the omnidirectional image isdisplayed. FIG. 33D is a view illustrating a display example of thepreview image. In the preview image according to the presentmodification, a balloon including the cut-out/distortion-corrected imageis not displayed, but the cut-out/distortion-corrected image is directlydisplayed in a superimposed manner on the omnidirectional image.Further, in the preview image, the cut-out/distortion-corrected image isdisposed at an end point of the flick operation.

As long as the preview image is configured to display simultaneously theomnidirectional image and cut-out/distortion-corrected image as in thepresent modification, the preview image is not limited to the exampledescribed in the above embodiments. For example, in the preview image,the omnidirectional image and cut-out/distortion-corrected image may bearranged in parallel.

Further, in the above embodiments, for the omnidirectional image, themasking frame is superimposed on the omnidirectional image so as topresent the masking area; while for the cut-out/distortion-correctedimage, the masking processing is actually performed for the masking areaso as to present the masking area. However, the present technology isnot limited to this. The masking area on the omnidirectional image maybe subjected to the masking processing so as to represent the maskingarea on the omnidirectional image, and the masking frame may besuperimposed on the cut-out/distortion-corrected image so as to presentthe masking area on the cut-out/distortion-corrected image.

Further, in the above embodiments, when specifying the masking area, theuser specifies two points of start point and end point on theomnidirectional image and sets, as the masking area on theomnidirectional image, a rectangle formed by vertical and horizontalsides and having the specified two points as diagonal vertexes. However,the present technology is not limited to this. For example, thefollowing procedure may be taken: specifying two points on theomnidirectional image to calculate, on the cut-out/distortion-correctedimage, a rectangular area formed by vertical and horizontal sides andhaving the specified two points as the two vertexes; performingtransform inverse to the cut-out/distortion correction processing forthe calculated rectangular area to calculate the masking area on theomnidirectional image; and superimposing the calculated masking area onthe omnidirectional image. In this procedure, the user specified the twopoints on the omnidirectional image; however, on the omnidirectionalimage, the rectangular having the specified two points as the diagonalvertexes is not displayed as the specified area to be processed, but adistorted area (which assumes a rectangle after the cut-out/distortioncorrection processing) is superimposed on the omnidirectional image.Further, it goes without saying that the masking area to be specified isnot limited to the rectangle.

Further, in the above embodiments, the masking area is specified byspecifying two points on the omnidirectional image. However, the presenttechnology is not limited to this. A size and a shape of the maskingarea may previously be determined, and in this case, the user mayspecify only one point for specifying the position of the masking areahaving a predetermined size and a predetermined shape on theomnidirectional image.

Further, in the above embodiments, when the masking area is specified onthe omnidirectional image, the cut-out/distortion-correction image onwhich the masking processing is performed for the masking area issynthesized with the omnidirectional image to thereby generate thepreview image. However, the present technology is not limited to this.For example, the following procedure may be taken: previously displayingthe cut-out/distortion-corrected image; specifying the masking area onthe previously displayed cut-out/distortion-corrected image to presentthe specified masking area on the cut-out/distortion-corrected image;synthesizing the omnidirectional image on which the masking processingis performed for the masking frame representing the masking area andcut-out/distortion-corrected image; and displaying the resultantsynthetic image.

Further, in the above embodiments, the examples in which the cut-outrange is previously determined and in which the cut-out range isdetermined based on the specification of the masking area have beendescribed; however, the cut-out range may be determined according toanother method. For example, the cut-out range may be determined byuser's specification of the cut-out range independent of the maskingarea specification. Further, a position of the cut-out range may befixed or movable. Further, a different cut-out range may be set everytime the user re-specifies the masking area after “canceling” thedisplayed preview image.

Further, in the above embodiments, one cut-out/distortion-correctedimage on which the masking processing is performed for the masking areaspecified on the omnidirectional image is synthesized with theomnidirectional image to generate the preview image; however, aplurality of the cut-out/distortion-corrected images on each of whichthe masking processing is performed for the masking area may bedisplayed in the preview image. That is, even where one masking area isspecified on the omnidirectional image, when the cut-out range set forthe masking area is different, the masking area on thecut-out/distortion-corrected image corresponding to each cut-out rangeis accordingly different. Thus, when different cut-out ranges are setfor the masking area, the preview image may display a plurality of thecut-out/distortion-corrected images, on each of which the maskingprocessing is performed for the masking area, corresponding to thedifferent cut-out ranges. Further, when a plurality of the masking areasis specified on the omnidirectional image, a preview image including aplurality of the cut-out/distortion-corrected images may be generated byperforming the same processing as above for each masking area.

Further, in the above embodiments, the masking processing is processingof blacking out the processing range for privacy protection; however,the present technology is not limited to this. For example, it ispossible to perform other processing for reducing an amount ofinformation, such as processing (mosaic processing) of reducing aresolution of the masking area, so as to protect privacy. Further, it ispossible to perform, as the masking processing, region of interest (ROI)processing of specifying an important range as a processing range andpreferentially allocating a code amount to the processing range, so asto preferentially assure image quality of the processing range even on alow-speed communication network.

Further, in the above embodiments, the omnidirectional image imagingsection images the omnidirectional image as the image having distortionusing a fish-eye lens, and the image display system acquires thisomnidirectional image; however, the image having distortion is notlimited to the omnidirectional image as long as it is an image havingdistortion that can be corrected.

3. Seventh Embodiment

The following describes a seventh embodiment of the present technology.FIG. 35 is a block diagram of a shooting system according to the seventhembodiment of the present technology. As illustrated in FIG. 35, ashooting system 107 includes a camera 170, a viewer 270, and an imageprocessor 370. The image processor 370 is connected with the camera 170and viewer 270.

The camera 170 is used to shoot an image to be subject to imageprocessing in the image processor 370 and to be displayed on the viewer270. The camera 170 is fixed to a predetermined position so as to shootan object to be shot.

The viewer 270 is a device for a user to view an image shot by thecamera 170 and subjected to image processing in the image processor 370.The viewer 270 may be, e.g., a personal computer or a mobile terminal.In a case where the camera 170 is a monitoring camera, the camera 170 isinstalled so as to be able to shot a place to be monitored, and theviewer 270 is used by a person who monitors the image.

The shooting system 370 is disposed between the camera 170 and viewer270 and connected thereto. The image processor 370 may be connected tothe camera 170 and viewer 270 by wire or wireless. Further, some or allof the constituent elements of the image processor 370 may be providedin the camera 170 or the viewer 270.

The camera 170 includes an omnidirectional imaging section 171. Theomnidirectional imaging section 171 is an imaging system having afish-eye lens which is a kind of a wide-angle lens. The omnidirectionalimaging section 171 is used to image a circular omnidirectional image.The omnidirectional image is the image having distortion imaged by afish-eye lens using a projection system, such as a stereographicprojection system, an equidistance projection system, an equisolid angleprojection system or an orthogonal projection system. Theomnidirectional imaging section 171 outputs the obtained omnidirectionalimage to the image processor 370. The image taken and output by theomnidirectional imaging section 171 may be a still image or a movingimage.

The viewer 270 includes an input section 272 and an image displaysection 271. The input section 272 performs various input operationsusing an input device such as a mouse and a touch panel. Particularly,in the present embodiment, the input section 272 performs input forposition specification and for range decision on the omnidirectionalimage shot by the camera 170.

The image display section 271 displays an image output from the imageprocessor 370. The image display section 271 may be, e.g., a liquidcrystal display panel. The image display section 271 displays a previewimage for a user to specify a processing range in a range specificationmode to be described later, while displays a normal image on which theprocessing range is subjected to predetermined processing in a normalmode to be described later.

The image processor 370 includes a display switching section 371, arange processing section 372, a range storage section 373, a rangesuperimposing section 374, a decision determination section 375, adistortion correction section 376, an image output section 377, a rangecalculation section 378, and a range specification section 379. Theimage processor 370 operates in a normal mode or a range specificationmode. The normal mode is a mode for displaying the image shot by thecamera 170 on the viewer 270, and the range specification mode is a modefor determining a processing range.

The display switching section 371 receives an input of theomnidirectional image shot by the omnidirectional imaging section 171and then determines an output destination of the omnidirectional image.The display switching section 371 outputs the input image to the rangeprocessing section 372 in the normal mode while outputs the input imageto the range superimposing section 374 in the range specification mode.The display switching section 371 outputs, to the image output section377, information indicating which one of the sections 374 and 371 isdetermined as the output destination.

The normal mode includes a plurality of display modes, such as a modefor displaying a panoramic image, a mode for displaying a plurality ofthe cut-out/distortion-corrected images, and a mode for displaying asingle cut-out/distortion-corrected image. The display switching section371 outputs, together with the input image, information related to thedisplay mode to the range processing section 372.

The range processing section 372 acquires the omnidirectional image fromthe display switching section 371 and uses information related to theprocessing range stored in the range storage section 373 to performimage processing for the processing range in the omnidirectional image.In the present embodiment, the range processing section 372 performs, asthe image processing, masking processing of blacking out the processingrange. Thus, the processing range corresponds to the masking area.

The range processing section 372 adopts a processing range correspondingto the display mode from the processing range stored in the rangestorage section 373. Further, when the omnidirectional image is directlydisplayed in a case where a plurality of processing rangescorresponding, respectively, to a plurality of display modes is storedin the range storage section 373, the range processing section 372 set,as the processing range, a range obtained by combining (AND processing)the plurality of processing ranges.

The range storage section 373 stores information related to theprocessing range to be subjected to image processing by the rangeprocessing section 372. In a case where the normal mode includes theplurality of display modes, the range storage section 373 stores theprocessing range for each display mode. The information related to theprocessing range is acquired from the decision determination section375.

The range storage section 373 may store the processing range as a set ofcoordinates of positions on an outer periphery (frame) of the processingrange on the omnidirectional image or may store the processing range asa set of functions representing curves constituting the outer periphery(frame) of the processing range.

FIGS. 36A and 36B are each a view illustrating the processing rangestored in the range storage section 373. FIG. 36A illustrates a casewhere the processing range is stored as a set of coordinates, and FIG.36B illustrates a case where the processing range is stored as a set offunctions. In FIG. 36A, the range storage section 373 stores coordinates(x0, y0), (x1, y1), . . . of all points (pixels) constituting the outerperiphery of the processing range. In FIG. 36B, the range storagesection 373 stores four functions f(x), g(x), h(x), and k(x)respectively representing curves constituting the outer periphery of theprocessing range.

The range superimposing section 374, decision determination section 375,range calculation section 378, and range specification section 379 areused in the range specification mode. The mode (range specification modeor normal mode) to be selected is specified by user operation throughthe input section 272 of the viewer 270, and the input information isgiven to the image processor 370.

The range superimposing section 374 acquires the omnidirectional imagefrom the display switching section 371 in the range specification mode,superimposes image information representing an undecided processingrange output from the decision determination section 375 on the acquiredomnidirectional image, and outputs the resultant image to the imageoutput section 377. In the present embodiment, the image informationrepresenting the processing range is a line segment surrounding theprocessing range. However, the present technology is not limited tothis, but the image information may be another information such as afilled processing range.

When the omnidirectional image is displayed as the preview image on theimage display section 271 of the viewer 270 in the range specificationmode, the range specification section 379 acquires the coordinates thatthe user specifies on the displayed omnidirectional image through theinput section 272 and outputs the acquired coordinates to the rangecalculation section 378.

FIGS. 37A and 37B are each a view for explaining the range specificationmade by the user. FIG. 37A is a view for explaining start of the rangespecification, and FIG. 37B is a view for explaining operation duringthe range specification or decision of the range specification. When theomnidirectional image as the preview image for the range specificationis displayed on the image display section 271, the user specifies theprocessing range on the displayed omnidirectional image. In the presentembodiment, the input section 272 is a device that moves a pointer usinga mouse, and the thus configured input section 272 is used to specifystart point and end point.

The user specifies two diagonal vertexes of a range that assumes arectangle after the distortion correction processing. Specifically, theuser uses the input section 272 to put a pointer over a portion to beset as the start point as illustrated in FIG. 37 followed by clicking,and the pointer is dragged to a portion to be set as the end point asillustrated in FIG. 37B. When the drag operation is terminated at theportion to be set as the end point, the pointer position at that time isset as the end point. With the above operation, the start point and endpoint are decided. During the drag operation, a processing rangecalculated from the start point and end point corresponding to aposition of the pointer being dragged is displayed in a superimposedmanner on the preview image, as illustrated in FIG. 37B. While thepointer is being dragged, the range specification section 379 outputscoordinates of the start point and coordinates of the end pointcorresponding to a position of the pointer being dragged to the rangecalculation section 378; after the termination of the drag operation,the range specification section 379 outputs coordinates of the startpoint and coordinates of the decided end point to the range calculationsection 378.

The range calculation section 378 uses the coordinates of the startpoint and end point acquired from the range specification section 379 tocalculate a range on the omnidirectional image that assumes a rectangleafter the distortion correction processing. A shape of the range on theomnidirectional image that assumes, after the distortion correctionprocessing, a rectangle having two diagonal vertex specified by therange specification section 379 varies depending on the distortioncorrection parameter, i.e., a distortion correction method (panoramiccorrection, cut-out/distortion correction, and the like) or a position(cut-out center) and an angle of view (zoom magnification) of the areato be subjected to the distortion correction in the cut-out/distortioncorrection.

Thus, for calculation of the shape on the omnidirectional image thatassumes the rectangle having the specified two diagonal vertexes,information of the parameter (hereinafter, referred to as “distortioncorrection parameter”) for the distortion correction to be performed forthe omnidirectional image is required. The distortion correctionparameter includes, e.g., a distortion correction method (panoramiccorrection, cut-out/distortion correction, and the like) and a position(cut-out center) and an angle of view (zoom magnification) of the areato be subjected to the distortion correction in the cut-out/distortioncorrection.

When the distortion method is the cut-out/distortion correction, and thecut-out position is previously fixed, the range calculation section 378may have information of the cut-out position. When there is a pluralityof the previously fixed cut-out positions, the range calculation section378 may have the information for each display range thereof. Such afixed cut-out position is used to monitor comings and goings through adoor, and the position to be subjected to the cut-out/distortioncorrection is fixed to a position corresponding to the door. When thedistortion correction parameter is specified by the user, the rangecalculation section 378 may acquire user's input to the input section272 concerning the distortion correction parameter from the rangespecification section 379 and determine the distortion correctionparameter based on the acquired input. The distortion correctionparameter is shared with the distortion correction section 376 and, inthe normal mode, the distortion correction section 376 performs thedistortion correction based on the distortion correction parameter.

The distortion correction parameter to be used for calculation of theprocessing range may be determined based on the specification of theprocessing range made by the user. FIGS. 38A and 38B are each a viewillustrating processing of determining a display range based on thespecification of the processing range. FIG. 38A is a view illustratingthe start point and end point of the processing range specified on theomnidirectional image, and FIG. 38B is a view illustrating a state wherethe processing range illustrated in FIG. 38A is displayed on an imageafter the cut-out/distortion correction.

As illustrated in FIG. 38A, when the two diagonal vertexes of theprocessing range that assumes a rectangle after the distortioncorrection processing are specified by the above-described procedure,the range calculation section 378 determines a cut-out position in thedistortion correction parameter such that the cut-out center ispositioned at a midpoint between the two vertexes. The cut-out viewangle (zoom magnification) may be fixed or determined according to adistance between the start point and end point. For example, it ispossible to set the distortion correction parameter such that a lineconnecting the start point and end point has a certain proportion to adiagonal line of the cut-out range. In the example of FIGS. 38A and 38B,the distortion correction parameter is determined such that the lineconnecting the start point and end point has 70% of the diagonal line ofthe cut-out range, and the midpoint between the start point and endpointis located at the center of the cut-out range. When the start point andend point are set as the two diagonal vertexes (corners) of the cut-outrange, the processing range and cut-out range coincide with each other.In this case, the range processing section 372 performs image processingfor the entire cut-out range.

After the distortion correction parameter has thus been determined, therange calculation section 378 acquires the coordinates of the startpoint and end point from the range specification section 379 and thenuses the distortion correction parameter and information of the startpoint and end point of the processing range to calculate a processingrange on the omnidirectional image. The processing range on theomnidirectional image is a range that assumes a rectangle after thedistortion correction processing performed based on the distortioncorrection parameter.

When a plurality of the display modes is set, the range calculationsection 378 calculates the range that assumes a rectangle after thedistortion correction processing using the distortion correctionparameters corresponding to respective display modes. FIG. 39 is a viewillustrating the processing range displayed in a superimposed manner onthe preview image. Hereinafter, the distortion correction parameter isassumed to include a first distortion correction parameter (for, e.g.,cut-out/distortion corrected image corresponding to an arbitrary partialrange) and a second distortion correction parameter (for, e.g.,panoramic image).

Upon acquisition of the information of the start point and end point,the range calculation section 378 calculates the processing range thatassumes a rectangle after the correction based on each of the first andsecond distortion correction parameters.

The range calculation section 378 calculates a first rectangle having,as the diagonal vertexes, start point and end point obtained bycoordinate transform based on the first distortion correction parameter.The range calculation section 378 performs, for the first rectangle,inverse transform of the coordinate transform that is previouslyperformed according to the first distortion correction parameter. Inthis case, the range calculation section 378 performs coordinatetransform using an inverse function of a transform function for acoordinate transform method indicated by the first distortion correctionparameter. As a result, the first rectangle assumes a distorted shape.The distorted shape corresponds to a first processing range CA1 on theomnidirectional image illustrated in FIG. 39.

The range calculation section 378 calculates a second rectangle having,as the diagonal vertexes, start point and end point obtained bycoordinate transform based on the second correction parameter. The rangecalculation section 378 performs, for the second rectangle, inversetransform of the coordinate transform that is previously performedaccording to the second distortion correction parameter. As a result,the second rectangle assumes a distorted shape. The distorted shapecorresponds to a first processing range CA2 on the omnidirectional imageillustrated in FIG. 39.

When the user has not decided the endpoint (for example, when the useris dragging the mouse), the decision determination section 375 outputsthe processing range calculated by the range calculation section 378 tothe range superimposing section 374 and when receiving a deciding action(for example, termination of the mouse drag operation) that the usermakes for the input section 272, the decision determination section 375stores the processing range that the range calculation section 378calculates at that time point in the range storage section 373.

The range superimposing section 374 receives an input image(omnidirectional image) from the display switching section 371 in therange specification mode, superimposes the processing range output fromthe decision determination section 375 on the omnidirectional image, andoutputs the resultant image to the image output section 377. When theplurality of processing ranges corresponding to the plurality of displayranges is calculated by the range calculation section 378 as describedabove, the range superimposing section 374 superimposes all theprocessing ranges on the omnidirectional image.

The distortion correction section 376 is used in the normal mode. In thenormal mode, the omnidirectional image input to the display switchingsection 371 is output to the range processing section 372, where imageprocessing is performed for the processing range stored in the rangestorage section 373, and the omnidirectional image that is partiallysubjected to the image processing is input to the distortion correctionsection 376.

The distortion correction section 376 performs the distortion correctionprocessing for the omnidirectional image (on which image processing isperformed for the processing range) input to generate adistortion-corrected image. When there is a plurality of display modes,i.e., distortion correction methods, and a plurality of processingranges corresponding to the plurality of display modes are stored in therange storage section 373, the range processing section 372 reads outthe processing range according to the display mode and performs imageprocessing for the read out processing range. When thecut-out/distortion correction processing is performed, the cut-outcenter may be different from the center of the processing range and, inthis case, the processing range is positioned offset from the center ofthe cut-out range and assumes a slightly distorted shape. Further, whenthe processing range is calculated based on a plurality of cut-outcenters, image processing should be performed for the processing rangecalculated based on a cut-out center closest to that used in thedistortion correction section 376.

Then, also in the distortion correction section 376, thecut-out/distortion correction processing is performed on the displayrange corresponding to the display mode. When the display mode is a modefor displaying a panoramic image, the cut-out/distortion-corrected imageis the panoramic image; while the display mode is a mode for displayinga cut-out arbitrary range, the cut-out/distortion-corrected image is animage obtained by cutting out the arbitrary partial range followed bythe distortion correction processing for the cut out range. Thecut-out/distortion-corrected image generated by the distortioncorrection section 376 is output to the image output section 377.

The image output section 377 outputs the image output from the rangesuperimposing section 374 to the image display section 271 of the viewer270 in the range specification mode, while outputs the image output fromthe distortion correction section 376 to the image display section 271of the viewer 270. The image output section 377 acquires, from thedisplay switching section 371, information indicating which one of theimage output from the range superimposing section 374 and image outputfrom the distortion correction section 376 is to be output, that is,whether the current mode is the range specification mode or normal mode.

The image display section 271 displays the image output from the imageoutput section 377 as the preview image in the range specification mode,and displays the image output from the image output section 377 as anormal image that is partially subjected to image processing.

FIGS. 40A and 40B are each a view illustrating an example of display onthe image display section 271 in the normal. FIG. 40A illustrates anexample in which the cut-out/distortion-corrected image obtained bycutting out the arbitrary partial range from the omnidirectional image,followed by the distortion correction processing for the cut out range,and FIG. 40B illustrates an example in which the panoramic imageobtained by developing a right half of the omnidirectional image in apanoramic manner.

As illustrated in FIG. 40A, the first processing range CA1 superimposedon the omnidirectional image in FIG. 39 assumes a rectangle on thecut-out/distortion-corrected image, and the rectangular processing rangeis subjected to the masking processing. Further, as illustrated in FIG.40B, the second processing range CA2 superimposed on the omnidirectionalimage in FIG. 39 assumes a rectangle on the panoramic image, and therectangular processing range is subjected to the masking processing.

The following describes operation of the image processor 370. FIG. 34 isa flowchart illustrating operation of the image processor 370. Thedisplay switching section 371 inputs the omnidirectional image from theomnidirectional imaging section 171 (step S341) and determines whetherthe current mode is the range specification mode or normal mode (stepS342).

When the current mode is the range specification mode, (“rangespecification mode” in step S342), the display switching section 371outputs the omnidirectional image (input image) to the rangesuperimposing section 374. Then, the omnidirectional image is suppliedto the image display section 271 through the image output section 377 tobe displayed thereon as the preview image, and information of the startpoint and end point for specifying the processing range input throughthe input section 272 using the preview image is input to the rangespecification section 379 (step S343).

The range calculation section 378 uses the information of the startpoint and endpoint input to the range specification section 379 andinformation of the display range to perform the coordinate transform ofthe start point and end point, calculates a rectangle having thecoordinate-transformed start point and end point as the diagonalvertexes, and performs, for the calculated rectangle, inverse transformof the previous coordinate transform, to thereby calculate theprocessing range (step S344).

The decision determination section 375 determines whether or not therange is decided (step S345), and when the range is not decided (NO instep S345), outputs a line segment representing an outer periphery ofthe processing range calculated by the range calculation section 378 tothe range superimposing section 374. The range superimposing section 374then superimposes (synthesizes) the line segment on the omnidirectionalimage input from the display switching section 371 (step S346). Theomnidirectional image on which the line segment representing theprocessing range is output to the image display section 271 through theimage output section 377 to be displayed thereon (step S347). FIG. 37illustrates an example of the displayed image.

The user determines whether the displayed processing range is acceptableor not while confirming the preview image on which the line segmentrepresenting the processing range and uses the input section 272 toperform action of deciding the processing range (terminating the mousedrag operation) or action of continuing the range specification(continuing the mouse drag operation). When the processing range is notdecided, and processing of steps S345, S346, and S347 terminates, theprocessing flow returns to step S343 where the user is prompted tospecify a new end point.

When the processing range is decided (YES in step S345), the rangedeciding section 375 outputs information of the processing range to therange storage section 373 so as to store the processing range in therange storage section 373 (step S348). After completion of theprocessing of step S348, the operation mode is shifted to the normalmode. In a case where the operation mode is shifted to the normal modethrough step S348 or where it is determined in step S342 that thecurrent mode is the normal mode, the range processing section 372applies the processing range input from the range storage section 373 tothe omnidirectional image input from the display switching section 371and performs image processing for the processing range (step S349).

The distortion correction section 376 acquires the omnidirectional imageon which the processing range is subjected to image processing by therange processing section 372, cuts out the display range from theomnidirectional image, and performs the distortion correction processingfor the cut-out display range to thereby generate thecut-out/distortion-corrected image (step S350). The image output section377 outputs the cut-out/distortion-corrected image generated by thedistortion correction section 376 to the image display section 271, andthe image display section 271 displays the cut-out/distortion-correctedimage (step S351). FIGS. 40A and 41B each illustrate an example of thedisplayed image.

As described above, according to the shooting system of the presentembodiment, when the user specifies, on the omnidirectional image, thetwo diagonal vertexes of the range to be set as the processing range,the range that will be a rectangular range having the specified twopoints as the two diagonal vertexes after the omnidirectional image issubjected to the distortion correction processing by the distortioncorrection section 376 is displayed on the omnidirectional image,thereby allowing the user to confirm, on the omnidirectional image, theprocessing range that assumes a rectangle after the omnidirectionalimage is subjected to the distortion correction processing. Further, theprocessing range corresponding to a position of the pointer during themouse drag operation is displayed, thereby making it possible to confirmwhether or not a user specified position falls within the processingrange when the current pointer position is decided as the end point.Further, when there are a plurality of the distortion correction methods(that is, when there are a plurality of display ranges corresponding tothe plurality of display modes), processing ranges that each assumes arectangle after the distortion correction processing corresponding tothe plurality of display modes can be confirmed on the omnidirectionalimage simultaneously. This makes it possible to confirm whether or notthe user specified position falls within the processing range in everydisplay mode.

The seventh embodiment of the present technology has thus beendescribed. Various modifications can be made to the above embodimentwithin the scope of the present technology. For example, although imageprocessing is performed for an inside of the processing range in theabove embodiment, the image processing may be performed for an outsideof the processing range. For example, the image processing is notperformed for the processing range that has been specified as a range tobe displayed, but the masking processing may be performed for asurrounding area thereof. Further, it is possible to perform (region ofinterest (ROI) processing of specifying an important range as theprocessing range and preferentially allocating a code amount to theprocessing range, so as to preferentially assure image quality of theprocessing range even on a low-speed communication network.

Further, in the above embodiment, the range to be cut out by thedistortion correction section 376 may be determined based on the rangespecified by the range specification section 379 and, in this case, theentire specified range may be set as the processing range; however, inthis case, this processing range may be set as the range to be subjectedto the cut-out processing by the distortion correction section 376. Inthis case, the range processing section 372 cuts out the processingrange as the image processing, and the distortion correction section 376performs the distortion correction processing for an image correspondingto the cut range acquired from the range storage section 373.

Further, in the above embodiment, the range calculation section 378calculates the processing range that assumes a rectangle after thedistortion correction processing; however, the processing range to beobtained after the distortion correction processing may have anothershape, such as an ellipse or a circle. Further, when the processingrange to be obtained after the distortion correction processing has ashape other than the rectangle, the user may specify three or morepoints as points for calculating the processing range. Further, when ashape and a size of the processing range are fixed, the user may specifyonly the position of the processing range (for example, user may specifyonly one point representing a center of the processing range).

Further, after the information of the processing range specified anddecided by the user is stored in the range storage section 373, the usermay use the input section 272 to perform operation for the storedprocessing range to deform the processing range on the preview image,that is, to change the specification of the processing range. Also inthis case, the processing range to be deformed is always deformed into ashape that assumes a predetermined shape such as a rectangle after thedistortion correction processing.

Further, in the above embodiment, the omnidirectional image imagingsection 171 images the omnidirectional image as the image havingdistortion using a fish-eye lens, and the image processor 370 acquiresthis omnidirectional image; however, the image having distortion is notlimited to the omnidirectional image as long as it is an image havingdistortion that can be corrected.

4. Eighth Embodiment

The following describes an eighth embodiment of the present technology.As described above, the omnidirectional image can be deformed into thepanoramic image. Further, it is possible to display thecut-out/distortion-corrected image obtained by cutting out a part of theomnidirectional image and perform correction processing for the cut outpart. In this case, by changing in real time the cut-out position, a PTZcamera function can be achieved. As described above, the omnidirectionalimage can be displayed in various modes. Further, in a video camera,there may be a case where image processing is performed for a specifiedarbitrary range. For example, the masking processing for privacyprotection is performed.

However, when the range specification is made using the method asdescribed in Patent Literature 2, that specifies an arbitrary range onthe image having distortion, a shape of the range changes depending onpresence/absence of the distortion correction processing, a differencein display mode, or an image cut-out position. This complicates therange specification to be conducted by a user, and a position of thespecified range is difficult to understand. Further, in the technologydescribed in Patent Literature 2, it is necessary to project a largenumber of coordinates before and after the distortion correctionprocessing and, accordingly, an amount of information to be processedbecomes extremely large.

Thus, in the present embodiment, a shooting system capable of enhancingeasiness of user's range specification operation and reducing aprocessing amount of information concerning the range shared before andafter the distortion correction processing will be described.

FIG. 42 is a block diagram illustrating a configuration of a shootingsystem 108 according to the eighth embodiment of the present technology.As illustrated in FIG. 42, an image processor 380 is connected with acamera 180 and a viewer 280 to constitute the shooting system 108.

The camera 180 is used to shoot an image to be subject to imageprocessing in the image processor 380 and to be displayed on the viewer280. The camera 180 is fixed to a predetermined position so as to shootan object to be shot.

The viewer 280 is a device for a user to view an image shot by thecamera 180 and subjected to image processing in the image processor 380.The viewer 280 may be, e.g., a personal computer or a mobile terminal.In a case where the camera 180 is a monitoring camera, the camera 180 isinstalled so as to be able to shot a place to be monitored, and theviewer 280 is used by a person who monitors the image.

The image processor 380 is disposed between the camera 180 and viewer280 and connected thereto. The image processor 380 may be connected tothe camera 180 and viewer 280 by wire or wireless. Further, some or allof the constituent elements of the image processor 380 may be providedin the camera 180 or the viewer 280.

The camera 180 includes an omnidirectional imaging section 181. Theomnidirectional imaging section 181 is an imaging system having afish-eye lens which is a kind of a wide-angle lens. The omnidirectionalimaging section 181 is used to image a circular omnidirectional image.The omnidirectional image is the image having distortion imaged by afish-eye lens using a projection system, such as a stereographicprojection system, an equidistance projection system, an equisolid angleprojection system or an orthogonal projection system. Theomnidirectional imaging section 181 outputs the obtained omnidirectionalimage to the image processor 380. The image taken and output by theomnidirectional imaging section 181 may be a still image or a movingimage.

The viewer 280 includes an input section 282 and an image displaysection 281. The input section 282 performs various input operationsusing an input device such as a mouse and a touch panel. Particularly,in the present embodiment, the input section 282 performs input forposition specification on the omnidirectional image shot by the camera180.

The image display section 281 displays an image output from the imageprocessor 380. The image display section 281 may be, e.g., a liquidcrystal display panel. The image display section 281 displays an imagefor a user to specify a processing range in a range specification timeto be described later, and displays an image on which the processingrange is subjected to predetermined processing at an image display timeto be described later.

The image processor 380 includes an image acquisition section 381, adistortion correction section 382, a range application section 383, animage processing section 384, a display setting section 385, a rangegeneration section 386, a position specification section 387, a positionstorage section 388, and a coordinate projection section 389. The imageprocessor 380 has two functions of creating, from an image shot by thecamera 180, an image to be displayed on the viewer 280 and specifyingthe processing range.

The image acquisition section 381 acquires the omnidirectional imageshot by the omnidirectional imaging section 181 of the camera 180.

The distortion correction section 382 performs distortion correctionprocessing for the omnidirectional image acquired by the imageacquisition section 381. Various distortion correction methods areassumed, and the correction method to be used is determined by adistortion correction parameter. FIG. 49A is a view illustrating anexample of a conventional omnidirectional image, FIG. 49B is a viewillustrating an example in which cut-out images corresponding to aplurality of cut-out ranges are displayed in a juxtaposed manner,together with the conventional omnidirectional image, and FIG. 49C is aview illustrating a display example of a panoramic image of theconventional omnidirectional image. For example, there are available amethod as illustrated in FIG. 49B that displays a plurality of rangesthat is cut out from the omnidirectional image of FIG. 49A and thensubjected to the correction processing (hereinafter, referred tocut-out/distortion correction processing) and a method as illustrated inFIG. 49C that displays panoramic images, one above the other, each with180-degree viewing angle generated from the omnidirectional image(hereinafter, referred to panoramic correction or panoramicdevelopment). In the present embodiment, the distortion correctionsection 382 performs the cut-out/distortion correction processing orpanoramic processing and outputs the resultant image. The distortioncorrection parameter includes the above distortion correction method anda parameter (for example, in the cut-out/distortion correctionprocessing, a cut-out position and size) required in the method.

The display setting section 385 sets the distortion correction parameterfor the distortion correction processing by the distortion correctionsection 382. When the distortion correction processing is not performed,the display setting section 385 does not set the distortion correctionparameter. The distortion correction parameter set by the displaysetting section 385 is supplied to the distortion correction section382, range generation section 386, and coordinate projection section389.

The range application section 383 acquires an image that is subjected tothe distortion correction processing from the distortion correctionsection 382, adds range information generated by the range generationsection 386 to the acquired image, and outputs the resultant image. Theadded range information is used as a processing range for arbitraryimage processing. When the range generation section 386 does notgenerate the range information, the range application section 383outputs directly the input image.

The image processing section 384 acquires, from the range applicationsection 383, the image added with the range information and performsarbitrary image processing for a range corresponding to the rangeinformation. The image processing may be applied to an inverse range ofthe range information. In the present embodiment, the image processingsection 384 performs, as the image processing, privacy maskingprocessing of blacking out the processing range. Thus, the processingrange corresponds to the masking area.

The position specification section 387 transmits coordinate informationon the display image based on input operation performed through theinput section 282. In the present embodiment, position informationcorresponds to a coordinate value of one point on the image to bedisplayed on the image display section 281.

The coordinate projection section 389 transforms, based on thedistortion correction parameter set by the display setting section 385,the coordinate information on the display image input from the positionspecification section 387 into coordinate information on theomnidirectional image.

The position storage section 388 stores the coordinate information(hereinafter, referred to as “position information”) on theomnidirectional image input from the coordinate projection section 389.In the present embodiment, the position storage section 388 stores onepoint in an orthogonal coordinate system on the omnidirectional image.The position information may use other coordinate system such as a polarcoordinate system and may be retained in the form of a positioninformation table that can be used in common between different displaysettings.

The range generation section 386 generates, based on the positioninformation stored in the position storage section 388 and distortioncorrection parameter set by the display setting section 385, rangeinformation representing a prescribed shape located at a specifiedposition. In the present embodiment, the range generation section 386transforms the position coordinates on the omnidirectional image storedin the position storage section 388 into a coordinate system on theimage that is subjected to the distortion correction processing based onthe distortion correction parameter and generates, as the rangeinformation, a circle having a fixed radius and centering on theobtained position coordinates.

The following describes operation of the image processor 380. Theoperation of the image processor 380 includes two operations of imagedisplay and range setting. First, the image display operation will bedescribed. FIG. 43 is a flowchart illustrating processing in the imagedisplay device according to the eighth embodiment of the presenttechnology. The image acquisition section 381 outputs theomnidirectional image input from the camera 180 to the distortioncorrection section 382 (step S431), and the distortion correctionsection 382 determines whether the distortion correction parameter to bereceived from the display setting section 385 is present or absent (stepS432).

When the distortion correction parameter is present, the distortioncorrection section 382 performs the distortion correction processingbased on the distortion correction parameter (step S433). When thedistortion correction parameter is absent, processing of step S433 isskipped, and the processing flow proceeds to step S434.

Then, it is determined whether the position information of the positionstorage section 388 is present or absent (step S434). The positioninformation of the position storage section 388 is generated when rangesetting operation to be described later is performed. When the rangesetting is performed, that is, when the position information is present,the range generation section 386 outputs the range informationcorresponding to the position information, and the range applicationsection 383 adds the range information to the image output from thedistortion correction section 382 (step S435). When the range setting isnot performed, or setting is reset, that is, when the range informationis absent, the addition of the range information is not made.

Here, details of generation of the range information will be described.FIG. 44 is a flowchart illustrating processing in the range generationsection according to the present embodiment. The range generationsection 386 acquires the position information from the position storagesection (step S441). In the present embodiment, the position informationcorresponds to coordinates of one point on the omnidirectional image.Then, the range generation section 386 determines whether the distortioncorrection parameter of the display setting section 385 is present orabsent (step S442).

When the distortion correction processing is present, the rangegeneration section 386 transforms the position information based on thedistortion correction parameter acquired from the display settingsection 385. In the present embodiment, the coordinates on theomnidirectional image is transformed into coordinates on the image thatis subjected to the distortion correction processing (step S443). Whenthe distortion correction processing is absent, the processing flowproceeds to step S444, skipping step S443. Subsequently, the rangegeneration section 386 generates the range information representing aprescribed shape with position information finally obtained set as abase point (step S444). In the present embodiment, a circle having aprescribed radius and centering on the finally obtained positioncoordinates after the distortion correction processing is formed as therange information. The position information finally obtained when thedistortion correction parameter is absent means coordinates on theomnidirectional image (position information retained in the positionstorage section 388).

Returning back to the flowchart of the image processor 380, the imageoutput from the range application section 383 is input to the imageprocessing section 384, and the image processing section 384 appliesarbitrary image processing to the processing range represented by therange information added to the image (step S436). The image that issubjected to the image processing is output to the image display section281 and displayed to the user (step S437).

Subsequently, the range setting operation to be performed in the imageprocessor 380 will be described. FIG. 45 is a flowchart illustratingprocessing of generating the position information in the presentembodiment. The position specification section 387 outputs the positioninformation on the image displayed on the image display section 281according to user's input information from the input section 282 (stepS451). The coordinate projection section 389 receives the settinginformation of the display setting section 385 and determines thepresence/absence of the distortion correction parameter (step S452).

When the distortion correction parameter is present in the settinginformation, the coordinate projection section 389 transforms, based onthe distortion correction parameter, the position information intocoordinates on the omnidirectional image before the distortioncorrection processing (step S453). When it is determined in step S452that the distortion correction parameter is absent, the positioninformation is unnecessary, so that step S453 is skipped, and theprocessing flow proceeds to step S454. Finally, the coordinates on theomnidirectional image are retained as the position information in theposition storage section (step S454).

FIG. 46 is a view illustrating an example of the image processing in thepresent embodiment. With reference to FIG. 46, an example in which therange specification is made on the image, and the image that issubjected to image processing is displayed will be described. Adistortion-corrected image 510 is an image including a plurality ofportions on an omnidirectional image 520 that are subjected to thecut-out/distortion correction processing by the distortion correctionsection 382. The user specifies a position 511 for the rangespecification on the distortion-corrected image 510 through the inputsection 282.

Then, the coordinate projection section 389 uses the distortioncorrection parameter for creation of the cut-out/distortion-correctedimage 510 from the omnidirectional image 520 to transform the specifiedposition 511 into coordinates 521 on the omnidirectional image 520.Subsequently, the position storage section 388 stores the coordinates521 obtained after transform.

Subsequently, assume that the user selects the panoramic development asthe distortion correction parameter. Then, the display setting section385 switches the distortion correction parameter from thecut-out/distortion correction to panoramic development. The distortioncorrection section 382 performs the panoramic development to create apanoramic development image 530. The range generation section 386 usesthe coordinates 521 read out from the position storage section 388 anddistortion correction parameter to calculate coordinates 531 on thepanoramic development image. Subsequently, the range generation section386 generates, as a range 532, a circle having a prescribed shape suchthat a point represented by the coordinates 531 is positioned at acenter of the circle.

Subsequently, the range application section 383 applies the generatedrange 532 to the panoramic development image 530. Further, the imageprocessing section 384 performs privacy processing for the range 532.Finally, the image display section 281 displays the image that issubjected to the privacy processing. In specifying the position 511, acircle having a prescribed shape may be displayed, centering on thecoordinates 511.

As described above, according to the image processor 380 of the presentembodiment, user's specification of only one point on thedistortion-corrected image allows achievement of image processing for anarbitrary range on the image. Further, at the time of image processing,the range to be subjected to the image processing always assumes thesame shape irrespective of the presence/absence of the distortioncorrection processing or the parameter type, thereby achievinguser-friendly image display.

The eighth embodiment of the present technology has thus been described.Various modifications can be made to the eighth embodiment within thescope of the present technology. For example, shapes other than thecircle may be used as the range. FIGS. 47A and 47B are views eachillustrating an example of the range information according to the eighthembodiment of the present technology. FIG. 47A illustrates a case wherea circle, a square, a regular triangle, and a regular hexagon are usedas the shape of the range. FIG. 47B illustrates a case where an ellipse,an oblong, and a rhombus are used as the shape of the range.

When the shape other than the circle is used as the range, it isnecessary to determine a direction of the shape. As a method ofdetermining the direction, there can be considered a method in which theuser specifies also the direction of the shape in performing theposition specification, in addition to the position thereof. FIG. 48 isa view illustrating an example of specification of the direction of theshape in the eighth embodiment of the present technology. A referencenumeral 710 is a displayed image that is subjected to the distortioncorrection processing by the distortion correction section 382, and theuser specifies a position 711 and a direction 712 as the rangespecification. Preferably, the user taps the position 711 on a touchpanel with his or her finger and slides the finger along the direction712. In the ellipse illustrated in FIG. 47A, the user may be prompted todetermine an aspect ratio of the ellipse. Further, the size of the shapecan be set by adjusting a release point of the finger being slid.

Alternatively, as the method for determining the direction, it can beconsidered a method that sets a certain direction on the image as adefault direction and sets the range according to the default direction.Specifically, the direction of the shape may be set such that an arrowin FIG. 47A coincides with a vertical direction of the image.

Further, in the above embodiment, the coordinates on the omnidirectionalimage are used as the position information; however, any information maybe used as long as it can be associated with one point on theomnidirectional image. For example, when distortion correction parameterand the coordinates on the distortion-corrected image may be set as theposition information. In this case, it is possible to associate thecoordinates on the distortion-corrected image with the coordinates onthe omnidirectional image by using the distortion-corrected parameter.Further, coordinates in an absolute space may be obtained by transformfor use as the position information.

Further, the size of the shape may be varied according to a zoommagnification of the distortion correction parameter. When the zoommagnification is high (when an object in the image is enlarged fordisplay), the size of the shape is set small; while, when the zoommagnification is low, the size of the shape is set large. Thus, evenwhen the zoom magnification is changed, it is possible to perform imageprocessing for the same target on the image.

Further, in the above embodiment, the omnidirectional image imagingsection 181 shoots the omnidirectional image using a fish-eye lens, andthe image processor 380 acquires this omnidirectional image; however,the image to be shot is not limited to the omnidirectional image as longas it is an image having distortion that can be corrected.

The first to eighth embodiments have been described as preferredembodiments of the present technology. It can be appreciated thatvarious modifications may be made to the above individual embodimentsand that and all such modifications within the spirit and scope of thepresent technology are included in the appended claims.

Further, a given embodiment and modifications of the given embodiment orother possibilities may be additionally or alternatively applied toanother embodiment wherever feasible.

INDUSTRIAL APPLICABILITY

As described above, the present technology has an effect of improvingappearance of the masked area included in the distortion-corrected imageobtained as a result of the distortion correction processing for theimage having distortion and is useful as an image processor or the likefor performing the distortion correction processing for the image havingdistortion, and applying the masking processing to a partial area of theimage having distortion.

REFERENCE SIGNS LIST

-   100 Camera-   11 Casing-   12 Dome cover-   13 Base-   14 Optical unit-   15 Main board-   16 Fish-eye lens-   17 Openable/closable cover-   18 Mounting bracket-   19 Lens holder-   20 Packing material-   21 Sensor board-   22 Through hole-   23 Fixing screw-   24 MOS holder-   25 MOS bracket-   26 Fixing screw-   28 Support post-   29 Fitting boss-   27 Coil spring-   101 to 103 Shooting system-   110, 120, 130 Camera-   111 Omnidirectional imaging section-   112 Masking data retaining section-   113 Masking execution section-   114 Cut-out parameter setting section-   115 Cut-out/distortion correction section-   126 Masking area shaping section-   1261 Corrected masking area generation section-   1262 Corrected masking area shaping section-   1263 Inverse transform/correction section-   137 Masking area distortion correction section-   138 Masking area shaping section-   210, 220, 230 Viewer-   211 Masking area detection section-   212 Masking area shaping section-   213 Output section-   104 to 106 Shooting system-   140, 150, 160 Camera-   141, 151, 161 Omnidirectional imaging section-   142, 152, 162 Masking area reception section-   143, 253, 163 Masking frame generation section-   144, 254, 164 Masking frame superimposing section-   145, 154 Masking execution section-   146, 155, 167 Cut-out/distortion correction section-   147, 156, 168 Synthesis section-   153 Cut-out range determination section-   165 Masking area distortion correction section-   166 Cut-out masking execution section-   240, 250, 260 Viewer-   241, 251, 261 Input section-   242, 252, 262 Masking area determination section-   243, 255, 263 Display switching section-   244, 256, 264 Display section-   107 Shooting system-   170 Camera-   171 Omnidirectional imaging section-   270 input section-   370 Image processor-   371 Display switching section-   372 Range processing section-   373 Range storage section-   374 Range superimposing section-   375 Decision determination section-   376 Distortion correction section-   377 Image output section-   378 Range calculation section-   379 Range specification section-   108 Shooting system-   180 Camera-   181 Omnidirectional imaging section-   280 Viewer-   281 Image display section-   282 Input section-   380 Image processor-   381 Image acquisition section-   382 Distortion correction section-   383 Range application section-   384 Image processing section-   385 Display setting section-   386 Range generation section-   387 Position specification section-   388 Position storage section-   389 Coordinate projection section

The invention claimed is:
 1. An image processing apparatus comprising: amasking area retainer that retains a masking area specified by a userfor an image, the image having distortion; a masking processor thatperforms masking processing for the image having distortion using themasking area of the masking area retainer; a distortion corrector thatcorrects the image that is subjected to the masking processing using acorrection parameter for correcting distortion of the image; and amasking area shaper that shapes the masking area of the corrected imageinto a predetermined shape; wherein the corrected image is displayedtogether with the image when the user specifies the masking area, thecorrected image is the image corrected by the distortion corrector, theimage displayed is the image having distortion that is subject to themasking processing by the masking processor.
 2. The image processingapparatus according to claim 1, further comprising a masking areadetector that detects the masking area of the corrected image correctedby the distortion corrector, wherein the masking area shaper shapes themasking area detected by the masking area detector into thepredetermined shape.
 3. The image processing apparatus according toclaim 1, wherein the masking area shaper shapes the masking area into arectangle surrounded by lines perpendicular and parallel to the image.4. The image processing apparatus according to claim 1, wherein themasking area shaper shapes the masking area such that a line that istransformed into a curve by distortion correction processing, by thedistortion corrector, is transformed into a straight line.
 5. The imageprocessing apparatus according to claim 1, wherein the image is an imagecut out from an omnidirectional image, and the masking area shapershapes the masking area according to a direction in which the image iscut out from the omnidirectional image.
 6. The image processingapparatus according to claim 1, wherein the distortion correctorlinearly interpolates source coordinates for each square formed by aplurality of pixels to fill in pixels of corresponding coordinates, andthe masking area shaper sets the square as the predetermined shape, thesquare including the masking area of the corrected image.
 7. The imageprocessing apparatus according to claim 6, wherein the masking areashaper sets a plurality of squares including the squares that includethe corrected masking area and combined into a rectangular shape, as amasking area after shaping.
 8. The image processing apparatus accordingto claim 1, further comprising an imager that inputs the image havingdistortion.
 9. The image processing apparatus according to claim 8,wherein the imager includes a fish-eye lens.
 10. The image processingapparatus according to claim 1, further comprising: a specifier that theuser uses to specify, on the image having distortion, a masking area tobe retained in the masking area retainer; and a masking area correctorthat corrects, by the correction parameter, the masking area specifiedby the specifier, wherein the masking area of the corrected image isdisplayed, in a superimposed manner, on the image having distortion whenuser specifies the masking area.
 11. An image processing apparatuscomprising: a masking area retainer that retains a masking area set foran image, the image having distortion; a masking area corrector thatcorrects the masking area of the masking area retainer using acorrection parameter for correcting the image; a masking area shaperthat shapes the masking area corrected by the masking area correctorinto a predetermined shape; an inverse transformer that performs inversetransform using the correction parameter for the masking area shaped bythe masking area shaper; a masking processor that performs maskingprocessing for the image using the masking area that is subjected to theinverse transform by the inverse transformer; and a distortion correctorthat corrects the image that is subjected to the masking processingusing the correction parameter.
 12. An image processing methodcomprising: acquiring an image, the acquired image having distortion;receiving a masking area, specified by user for the acquired image;performing masking processing for the image using the masking area;correcting the image that is subjected to the masking processing using acorrection parameter for correcting distortion of the image; shaping themasking area of the corrected image into a predetermined shape, anddisplaying the corrected image together with the image when the userspecifies the masking area, the corrected image being corrected by thecorrection parameter, displayed the image is the image having distortionthat is subject to the masking processing.
 13. An image processingapparatus comprising: a masking area receiver that receives a maskingarea corresponding to a partial area of an image, the image havingdistortion, the masking area being specified in response to user input;a masking processor that generates, based upon the masking area receivedby the masking area receiver, a masked image by performing maskingprocessing for the masking area of the image; and a distortion correctorthat generates a corrected image by correcting at least distortion ofanother partial area, including the masking area, of the masked imagegenerated by the masking processor, wherein the corrected image isdisplayed, on a display together with the image having distortion, whenuser designates the masking area.
 14. The image processing apparatusaccording to claim 13, wherein the corrected image is displayed on thedisplay such that the corrected image is superimposed on the imagehaving distortion.
 15. The image processing apparatus according to claim13, further comprising: a masking area indicator that indicates themasking area, received by the masking area receiver, on the image havingdistortion, wherein the corrected image is displayed, on the display,together with the image having distortion that is indicated by themasking area indicator.
 16. The image processing apparatus according toclaim 13, wherein when the corrected image is displayed together withthe image having distortion, and upon receiving a predetermined userinput, a display switch removes the corrected image from beingdisplayed.
 17. The image processing apparatus according to claim 13,further comprising: a display switch that changes a displayed image onthe display from combination of images, the combination of images beinga combination of the image having distortion with the corrected image,to the image having distortion, upon receiving a predetermined userinput.
 18. The image processing apparatus according to claim 13, whereinthe corrected image comprises a clipped image having a rectangular shapeand a smaller area than an area of the image having distortion.
 19. Theimage processing apparatus according to claim 13, wherein a framedisplayed on the image is moved in response to user input, when aposition of the frame is determined by the user input, the correctedimage, showing at least an area corresponding to the determinedposition, is displayed on the display together with the image havingdistortion.